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(54) METHOD OF PROCESSING, TRANSMITTING AND RECEIVING DYNAMIC IMAGE DATA AND 
V APPARATUS THEREFOR 

(57) A reception control section 1 1 for receiving the 
information including data and its transmission format «- * «- * 
information from a memory or communication channel, 
a separating section 12 for analyzing and separating 
received information, a transmitting section 13 for trans- 
mitting information to a memory or transmission chan- 
nel, a video extending section 14 for extending a video, 
and video-extension control section 15 control the 
processing state of said video extending section 14 for 
extending at least one or more videos and a video syn- 
thesizing apparatus constituted with a video synthesiz- 
ing section 16 for synthesizing videos in accordance 
with extended information, an output section 17 for out- 
putting a synthesized result, and a terminal control sec- 
tion 18 for controlling the above means makes it 
possible to synthesize a plurality of videos at the same 
time and correspond to a dynamic change of transmis- 
sion format information. 
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Description 

Technical Field 

[0001] The present invention relates to audio-video s 
transmitter and audio-video receiver, data-processing 
apparatus and method, waveform-data-transmitting 
method and apparatus and waveform-data-receiving 
method and apparatus, and video-transmitting method 
and apparatus and video-receiving method and appara- 10 
tus. 

Background Art 

[0002] There has been an apparatus which satisfies is 
the sense of real existence that a counterpart is present 
in front of you and aims at realistic picture communica- 
tion by extracting, for example, a person's picture out of 
the scenery picture of a space in which you are present 
and superimposing the person's picture, a person's pic- 20 
ture sent from the counterpart, and the picture of a vir- 
tual space to be displayed commonly with a previously- 
stored counterpart on each other and displaying them 
(Japanese Patent Publication No. 4-24914). 
[0003] Particularly, in the case of the prior art, inven- 25 
tions concerned with acceleration for performing picture 
synthesis and a method for reducing memories are 
made (e.g. Official gazette of Japanese Patent Publica- 
tion No. 5-46592: Picture synthesizer). 
[0004] Though a communication system using picture 30 
synthesis for synthesizing two-dimensional static pic- 
tures or three-dimensional CG data has been proposed 
by the prior art specific discussion on a method for real- 
izing a system for simultaneously synthesizing a plural- 
ity of video (picture) and a plurality of audio and 35 
displaying them has not been performed from the fol- 
lowing viewpoints. 

[0005] That is, there has been a problem that no spe- 
cific discussion has been performed from the following 
viewpoints: 40 

(A1) a method for transmitting (communicating and 
broadcasting) and controlling pictures and audio 
under the environment in which data and control 
information (information transmitted by a packet dif- 45 
ferent from that of data to control the processing of 
terminal side) are independently transmitted by 
using a plurality of logical transmission lines con- 
structed by software on one real transmission line 
or more; so 
(A2) a method for dynamically changing header 
information (corresponding to data control informa- 
tion of the present invention) to be added to data for 
a picture or audio to be transmitted; 
(A3) a method for dynamically changing header 55 
information (corresponding to transmission control 
information of the present invention) to be added for 
transmission; 



(A4) a method for transmitting information by 
dynamically multiplexing and separating a plurality 
of logical transmission lines; 
(A5) a method for transmitting pictures and audio 
considering the read and rise periods of program or 
data; and 

(A6) a method for transmitting pictures and audio 
considering zapping. 

[0006] However, the method for changing encoding 
systems and a method of discussing data in frames in 
accordance with the frame type of a picture have been 
proposed so far as a method for dynamically adjusting 
the amount of data to be transmitted to a network (H. 
Jinzenji and T. Tajiri. A study of distributive-adaptive- 
type VOD system, D-81, System Society of Institute of 
Electronics, information and Communication Engineers 
(IEICE) (1995)). 

[0007] A dynamic throughput scalable algorithm capa- 
ble of providing a high-quality video under a restricted 
processing time is proposed as a method for adjusting 
throughput at the encoder side (T. Osako, Y Yajima, H. 
Kbdera, H. Watanabe, K. Shimamura: Encoding of soft- 
ware video using a dynamic throughput scalable algo- 
rithm, Thesis Journal of IEICE, D-2, Vol. 80-D-2, No. 2, 
pp. 444-458(1997)). 

[0008] Moreover, there is an MPEG1/MPEG2 system 
as an example of realizing synchronous reproduction of 
video and audio. 

(B1) The conventional method for discussing a pic- 
ture correspondingly to the frame type of the video 
has a problem that it is difficult to preponderantly 
reproduce an important scene cut synchronously 
with audio by handling a plurality of video streams 
or a plurality of audio streams and reflecting the 
intention of an editor because the grading of the 
information which can be handled is in a single 
stream. 

(B2) Moreover, it must be possible that a decoder 
decodes every supplied bit stream because it is a 
prerequisite that MPEG1/MPEG2 is realized by 
hardware. Therefore, it is a problem how to corre- 
spond to the case of exceeding the throughput of 
the decoder. 

[0009] Moreover, to transmit video, there have been 
some systems including a system such as H. 261 (ITU- 
T Recommendation H. 261 -Video codec for audio-vis- 
ual services at px 64) and they have been mounted by 
hardware. Therefore, the case has not occurred that 
decoding is not completed within a designated time 
because of considering the upper limit of a necessary 
performance when designing hardware. 
[0010] The above-designated time denotes a time 
required to transmit a bit stream obtained by coding a 
sheet of video. If decoding is not completed within the 
time, an extra time becomes a delay. If the delay is accu- 
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mulated, the delay from the transmitting side to the 
receiving side increases and the system cannot be used 
as a video telephone. This state must be avoided. 
[0011] Moreover, when decoding cannot be com- 
pleted within a designated time because a communica- s 
tion counterpart generates an out-of-spec bit stream, a 
problem occurs that a video cannot be transmitted. 
[001 2] The above problem occurs not only for a video 
but also for audio data. 

[001 3] However, in recent years, because the network 10 
environment formed by personal computers (PCs) has 
been arranged as the result of spread of internet and 
ISDN, the transmission rate has been improved and it 
has been possible to transmit a video by using PCs and 
a network. Moreover, requests for transmission of video 15 
by users have been rapidly increased. Furthermore, a 
video can be completely decoded by software because 
CPU performances have been improved. 
[0014] However, because the same software can be 
executed by personal computers different in structure 20 
such as a CPU, bus width, or accelerator, it is difficult to 
previously consider the upper limit of a necessary per- 
formance and therefore, a problem occurs that a picture 
cannot be decoded within a designated time. 
[001 5] Moreover, when coded data for a video having 25 
a length exceeding the throughput of a receiver is trans- 
mitted, coding cannot be completed within a designated 
time. 

Problem (C1): Decreasing a delay by decoding a 30 
picture within a designated time. 

When inputting a video as the waveform data of 
claim CI of the present invention or outputting a 
video as the waveform data of claim C7 of the 
present invention as means for solving the problem 35 
1 , a problem may be left that the substantial work- 
ing efficiency of a transmission line is lowered 
because a part of a transmitted bit stream is not 
used. Moreover, there are some coding systems 
that generate a present decoded video in accord- 40 
ance with a last decoded picture (e.g. P picture). 
However, because the last decoded picture is not 
completely restored by the means for solving the 
problem 1, there is a problem that deterioration of 
'the picture quality irtfluentialiy increases as time 45 
passes. 

Problem (C2): In the case of the means for solving 
the problem 1, the substantial working efficiency of 
a transmission line is lowered. Moreover, picture- 
quality deterioration is spread. so 

Furthermore, in the case of mounting by soft- 
ware, the frame rate of a picture is determined by 
the time required for one-time coding. Therefore, 
when the frame rate designated by a user exceeds 
the throughput of a computer, it is impossible to cor- ss 
respond to the designation. 
Problem (C3) : When the frame rate designated by a 
user exceeds the throughput of a computer, it is 



impossible to correspond to the designation. 
Disclosure of the Invention 

[001 6] When considering the problems (A1 ) to (A6) of 
the first prior art, it is an object of the present invention 
to provide an audio-video transmitter and audio-video 
receiver and data-processing apparatus and method in 
order to solve at least any one of the problems. 
[001 7] Moreover, when considering the problems (B1 ) 
and (B2) of the second prior art, it is another object of 
the present invention to provide data-processing appa- 
ratus and method in order to solve at least one of the 
problems. 

[0018] Furthermore, when considering the problems 
(C1) to (C3) of the last prior art, it is still another object 
of the present invention to provide waveform-data- 
receiving method and apparatus and waveform-data- 
transmitting method and apparatus, and video-transmit- 
ting method and apparatus and video-receiving method 
and apparatus in order to solve at least one of the prob- 
lems. 

10019] The present invention according to claim 1 is 
an audio-video transmitting apparatus comprising 

transmitting means for transmitting the content con- 
cerned with a transmitting method and/or the struc- 
ture of data to be transmitted or an identifier 
showing the content as transmission format infor- 
mation through a transmission line same as that of 
the data to be transmitted or a transmission line dif- 
ferent from the data transmission line; wherein 
said data to be transmitted is video data and/or 
audio data. 

[0020] The present invention according to claim 2 is 
the audio-video transmitting apparatus according to 
claim 1 . wherein said transmission format information is 
included in at least one of data control information 
added to said data to control said data, transmission 
control information added to said data to transmit said 
data, and information for controlling the processing of 
the terminal side. 

[0021] The present invention according to claim 3 is 
the audio-video transmitting apparatus according to 
claim 2, wherein at least one of said data control infor- 
mation, transmission control information, and informa- 
tion for controlling the processing of said terminal side is 
dynamically changed. 

[0022] The present invention according to claim 4 is 
the audio-video transmitting apparatus according to 
claim 3, wherein said data is divided into a plurality of 
packets, and said data control information or said trans- 
mission control information is added not only to the 
head packet of said divided packets but also to a middle 
packet of them. 

[0023] The present invention according to claim 5 is 
the audio-video transmitting apparatus according to 
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claim 1, wherein an identifier showing whether to use 
timing information concerned with said data as informa- 
tion showing the reproducing time of said data is 
included in said transmission format information. 
[0024] The present invention according to claim 6 is 5 
the audio-video transmitting apparatus according to 
claim 1 , wherein said transmission format information is 
the structural information of said data and a signal 
which is output from a receiving apparatus receiving the 
transmitted structural information of said data and which 
can be received is confirmed and thereafter, said trans- 
mitting means transmits corresponding data to said 
receiving apparatus. 

[0025] The present invention according to claim 7 is 
the audio-video transmitting apparatus according to 
claim 1, wherein said transmission format information 
include (1) an identifier for identifying a program or data 
to be used by a receiving apparatus later and (2) at least 
one of a flag, counter, and timer as information for 
knowing the point of time in which said program or data 
is used or the term of validity for using said program or 
data. 

[0026] The present invention according to claim 8 is 
the audio-video transmitting apparatus according to 
claim 7, wherein said point of time in which said pro- 
gram or data is used is transmitted as transmission con- 
trol information by using a transmission serial number 
for identifying a transmission sequence or as informa- 
tion to be transmitted by a packet different from that of 
data to control terminal-side processing. 
[0027] The present invention according to claim 9 is 
the audio-video transmitting apparatus according to 
claim 2 or 3, wherein storing means for storing a plural- 
ity of contents concerned with said transmitting method 
and/or said structure of data to be transmitted and a plu- 
rality of its identifiers are included, and said identifier is 
included in at least one of said data control information, 
transmission control information, and information for 
controlling terminal-side processing as said transmis- 
sion format information. 

[0028] The present invention according to claim 10 is 
the audio-video transmitting apparatus according to 
claim 2 or 3, wherein storing means for storing a plural- 
ity of contents concerned with said transmitting method 
and/or said structure of data to be transmitted are 
included, and said contents are included in at least one 
of said data control information, transmission control 
information, and information for controlling terminal-side 
processing as said transmission format information. 
[0029] The present invention according to claim 11 is 
the audio-video transmitting apparatus according to 
claim 1, 2, or 3, wherein a default identifier showing 
whether to change the contents concerned with said 
transmitting method and/or structure of data to be trans- 
mitted is added. 

[0030] The present invention according to claim 12 is 
the audio-video transmitting apparatus according to 
claim 9, 10, or 11 , wherein said identifier or said default 



identifier is added to a predetermined fixed-length 
region of information to be transmitted or said predeter- 
mined position. 

[0031] The present invention according to claim 13 is 
an audio-video receiving apparatus comprising: receiv- 
ing means for receiving said transmission format infor- 
mation transmitted from the audio-video transmitting 
apparatus of any one of claims 1 to 12; and transmitted- 
information interpreting means for interpreting said 
received transmission-format information. 
[0032] The present invention according to claim 14 is 
the audio-video receiving apparatus according to claim 
13, wherein storing means for storing a plurality of con- 
tents concerned with said transmitting method and/or 
said structure of data to be transmitted and a plurality of 
its identifiers are included, and the contents stored in 
said storing means are used to interpret said transmis- 
sion format information. 

[0033] The present invention according to claim 15 is 
an audio-video transmitting apparatus comprising: infor- 
mation multiplexing means for controlling start and end 
of multiplexing the information for a plurality of logical 
transmission lines for transmitting data and/or control 
information is included; wherein, not only said data 
and/or control information multiplexed by said informa- 
tion multiplexing means but also control contents con- 
cerned with start and end of said multiplexing by said 
information multiplexing means are transmitted as mul- 
tiplexing control information, and said data includes 
video data and/or audio data. 

[0034] The present invention according to claim 16 is 
the audio-video transmitting apparatus according to 
claim 15, wherein it is possible to select whether to 
transmit said multiplexing control information by arrang- 
ing said information without multiplexing it before said 
data and/or control information or transmit said multi- 
plexing control information through a transmission line 
different from the transmission line for transmitting said 
data and/or control information. 
[0035] The present invention according to claim 1 7 is 
an audio-video receiving apparatus comprising: receiv- 
ing means for receiving said multiplexing control infor- 
mation transmitted from the audio-video transmitting 
apparatus of claim 15 and said multiplexed data and/or 
control information; and separating means for separat- 
ing said multiplexed data and/or control information in 
accordance with said multiplexing control information. 
[0036] The present invention according to claim 1 8 is 
an audio-video receiving apparatus comprising: main 
looking-listening means for looking at and listening to a 
broadcast program; and auxiliary looking-listening 
means for cyclically detecting the state of a broadcast 
program other than the broadcast program looked and 
listened through said main looking-listening means; 
wherein said detection is performed so that a program 
and/or data necessary when said broadcast program 
looked and listened through said main looking-listening 
means is switched to other broadcast program can be 
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smoothly processed, and 

said data includes video data and/or audio data. 
[0037] The present invention according to claim 1 9 is 
the audio-video transmitting apparatus according to 
claim 1, wherein priority values can be changed in 
accordance with the situation by transmitting the offset 
value of information showing the priority for processing 
of said data. 

[0038] The present invention according to claim 20 is 
an audio-video receiving apparatus comprising: receiv- 
ing means for receiving encoded information to which 
the information concerned with the priority for process- 
ing under an overload state is previously added; and pri- 
ority deciding means for deciding a threshold serving as 
a criterion for selecting whether to process an object in 
said information received by said receiving means; 
wherein 

the timing for outputting said received informa- 
tion is compared with the elapsed time after start of 
a processing or the timing for decoding said received 

information is compared with the elapsed time after 
start of processing to change said threshold in accord- 
ance with the comparison result, and video data and/or 
audio data are or is included as said encoding object. 
[0039] The present invention according to claim 21 is 
the audio-video receiving apparatus according to claim 
20, wherein retransmission-request-priority deciding 
means for deciding a threshold serving as a criterion for 
selecting whether to request retransmission of some of 
said information not received because it is lost under 
transmission when it is necessary to retransmit said 
information is included, and 

said decided threshold is decided in accordance 
with at least one of the priority controlled by said priority 
deciding means, retransmission frequency, lost factor of 
information, insertion interval between in-frame- 
encoded frames, and grading of priority. 
[0040] The present invention according to claim 22 is 
an audio-video transmitting apparatus comprising: 
retransmission-priority deciding means for deciding a 
threshold serving as a criterion for selecting whether to 
request retransmission of some of said information not 
received because it is lost under transmission when 
retransmission of said unreceived information is 
requested is included, wherein said decided threshold is 
decided in accordance with at least one of the priority 
controlled by the priority deciding means of said audio- 
video receiving apparatus of claim 20. retransmission 
frequency, lost factor of information, insertion interval 
between in-frame-encoded frames, and grading of prior- 
ity. 

[0041] The present invention according to claim 23 is 
an audio-video transmitting apparatus for transmitting 
said encoded information by using the priority added to 
said encoded information and thereby thinning it when 
(1) an actual transfer rate exceeds the target transfer 
rate of information for a video or audio or (2) it is 
decided that writing of said encoded information into a 
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transmitting buffer is delayed as the result of comparing 
the elapsed time after start of transmission with a period 
to be decoded or output added to said encoded informa- 
tion. 

5 [0042] The present invention according to claim 25 is 
a data processing apparatus comprising: receiving 
means for receiving a data series including (1) time- 
series data for audio or video, (2) an inter-time-series- 
data priority showing the priority of the processing 

10 between said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said time- 
series data value to show the processing priority 
between divided data values; and data processing 
means for performing Processing by using said inter- 

is time-series-data priority and said in-time-series-data 
priority together when pluralities of said time-series- 
data values are simultaneously present. 
[0043] The present invention according to claim 27 is 
a data processing apparatus comprising: receiving 

20 means for receiving a data series including (1) time- 
series data for audio or video, (2) an inter-time-series- 
data priority showing the priority of the processing 
between said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said time- 
rs series data value to show the processing priority 
between divided data values; and data processing 
means for distributing throughput to each of said time- 
series-data values in accordance with said inter-time- 
series-data priority and moreover, adaptively deteriorat- 

30 ing the processing quality of the divided data in said 
time-series data in accordance with said in-time-series- 
data priority so that each of said time-series-data values 
is kept within said distributed throughput. 
[0044] The present invention according to claim 29 is 

35 a data processing apparatus characterized by, when an 
in-time-series-data priority for a video is added every 
frame of said video and said video for each frame is 
divided into a plurality of packets, adding said in-time- 
series-data priority only to the header portion of a 

40 packet for transmitting the head portion of a frame of 
said video accessible as independent information. 
[0045] The present invention according to claim 31 is 
the data processing apparatus according to any one of 
claims 25, 27, and 29, wherein said in-time-series-data 

45 priority is described in the header of a packet to perform 
priority processing, 

[0046] The present invention according to claim 33 is 
the data processing apparatus according to any one of 
claims 25, 27, and 29, wherein the range of a value 

so capable of expressing said in-time-series-data priority is 
made variable to perform priority processing. 
[0047] The present invention according to claim 34 is 
a data processing method comprising the steps of: 
inputting a data series including time-series data for 

55 audio or video and an inter-time-series-data priority 
showing the processing priority between said time- 
series data values; and 

processing priorities by using said inter-time- 
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series-data priority as the value of a relative or absolute 
priority. 

[0048] The present invention according to claim 36 is 
a data processing method comprising the steps of: clas- 
sifying time-series data values for audio or video; input- s 
ting a data series including said time-series data and a 
plurality of in-time-series-data priorities showing the 
processing priority between said classified data values; 
and processing priorities by using said in-time-series- 
data priority as the value of a relative or absolute prior- w 
tty. 

[0049] Moreover, to solve the problem (C1), the 
present invention is characterized by: 

inputting, for example, a video as waveform data in 15 
accordance with the waveform-data-transmitting 
method of claim 63; or 

outputting, for example, a video as waveform data 
in accordance with the waveform-data-receiving 
method of claim 69. 20 

[0050] Moreover, to solve the problem (C2), the 
present invention is characterized by: 

(d) outputting the execution time of each group 2s 
obtained through estimation in accordance with the 
waveform-data-receiving method of claim 69;or 

(d) inputting a data string constituted with the exe- 
cution time of each group; and 

(e) computing the execution frequency of each 30 
group for completing decoding within a time 
required to transmit a code length determined by 
the designation of a rate controller or the like in 
accordance with each execution time of the receiv- 
ing means in accordance with the wave-data-trans- 35 
mitting method of claim 63. 

[0051] Furthermore, to solve the problem (C3), the 
present invention is characterized by: 

40 

(d) estimating the execution time of each group in 
accordance with the processing time required to 
encode a video and each execution frequency out- 
put by counting means; and 

(e) estimating the processing time required to 45 
encode a video by using the above execution time 
and computing the execution frequency of each 
group in which the processing time does not 
exceed a time usable to process one sheet of pic- 
ture determined by a frame rate given as the desig- so 
nation of a user in accordance with the waveform- 
data-transmitting method of claim 67. 

[0052] The present invention has the above structure 
to obtain the execution frequency of indispensable ss 
processing and that of dispensable processing, transmit 
the execution frequencies to the receiving side, and 
estimate the time required for each processing in 



accordance with the execution frequencies and the 
decoding time. 

[0053] By reducing each execution frequency of dis- 
pensable processing so that the time required for 
decoding becomes shorter than a designated time in 
accordance with the estimated time of each processing, 
it is possible to control the decoding time to the desig- 
nated time or shorter and keep a delay small. 
[0054] Claims 67 and 73 are mainly listed as the 
inventions for solving the problem (C1). 
[0055] Moreover, it is possible to set the decoding exe- 
cution time to a value equal to or less than a designated 
time by transmitting the execution time of indispensable 
processing and that of dispensable processing esti- 
mated by the receiving side to the transmitting side and 
determining each execution frequency at the transmit- 
ting side in accordance with each execution time. 
[0056] Claims 75 and 77 are mainly listed as the 
inventions for solving the problem (C2). 
[0057] Moreover, it is possible to set the encoding esti- 
mation time to a value equal to or less than a user des- 
ignated time by estimating the execution time of 
indispensable processing and that of dispensable 
processing and determining each execution frequency 
in accordance with each execution time and the user 
designated time determined by a frame rate designated 
by a user. 

[0058] Claim 79 is mainly listed as the invention for 
solving the problem (C3). 

Brief Description of the Drawings 

[0059] 

Figure 1 is a schematic block diagram of the audio- 
video transceiver of an embodiment of the present 
invention; 

Figure 2 is an illustration showing a reception con- 
trol section and a separating section; 
Figure 3 is an illustration shoving a method for 
transmitting and controlling video and audio by 
using a plurality of logical transmission lines; 
Figure 4 is an illustration showing a method for 
dynamically changing header information added to 
the data for a video or audio to be transmitted; 
Figures 5(a) and 5(b) are illustrations showing a 
method for adding AL information; 
Figures 6(a) to 6(d) are illustrations showing exam- 
ples of a method for adding AL information; 
Figure 7 is an illustration showing a method for 
transmitting information by dynamically multiplexing 
and separating a plurality of logical transmission 
lines; 

Figure 8 is an illustration showing a procedure for 
transmitting a broadcasting program; 
Figure 9(a) is an illustration showing a method for 
transmitting a video or audio considering the read 
and rise time of program or data when the program 
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or data is present at a receiving terminal; 

Figure 9(b) is an illustration showing a method for 

transmitting a video or audio considering the read 

and rise time of program or data when the program 

or data is transmitted ; 5 

Figure 10(a) is an illustration showing a method for 

corresponding to zapping; 

Figure 10(b) is an illustration showing a method for 

corresponding to zapping; 

Figure 11(a) is an illustration showing a specific 10 
example of the protocol to be actually transferred 
between terminals; 

Figure 11(b) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; 15 
Figure 12 is an illustration showing a specific exam- 
pie of the protocol to be actually transferred 
between terminals; 

Figure 13(a) is an illustration showing a specific 
example of the protocol to be actually transferred 20 
between terminals; 

Figure 13(b) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; 

Figure 13(c) is an illustration showing a specific 25 
example of the protocol to be actually transferred 
between terminals; 

Figure 14 is an illustration showing a specific exam- 
ple of the protocol to be actually transferred 
between terminals; 30 
Figure 15 is an illustration showing a specific exam- 
ple of the protocol to be actually transferred 
between terminals; 

Figure 16(a) is an illustration showing a specific 
example of the protocol to be actually transferred 35 
between terminals; 

Figure 16(b) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; 

Figure 1 7 is an illustration showing a specific exam- 40 
pie of the protocol to be actually transferred 
between terminals; 

Figure 18 is an illustration showing a specific exam- 
ple of the protocol to be actually transferred 
between terminals; as 
Figure 19(a) is an illustration showing a specific 
example of the protocol to be actually transferred 
between terminals; 

Figure 19(b) is an illustration showing a specific 
example of the protocol to be actually transferred so 
between terminals; 

Figures 20(a) to 20(c) are block diagrams of dem- 
onstration systems of CGD of the present invention; 
Figure 21 is an illustration showing a method for 
adding a priority under overload at an encoder; ss 
Figure 22 is an illustration describing a method for 
deciding a priority at a receiving terminal under 
overload; 
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Figure 23 is an illustration showing temporal 
change of priorities; 

Figure 24 is an illustration showing stream priority 
and object priority; 

Figure 25 is a schematic block diagram of a video 
encoder and a video decoder of an embodiment of 
the present invention; 

Figure 26 is a schematic block diagram of an audio 
encoder and an audio decoder of an embodiment of 
the present invention; 

Figures 27(a) and 27(b) are illustrations showing a 
priority adding section and a priority deciding sec- 
tion for controlling the priority of processing under 
overload; 

Figures 28(a) to 28(c) are illustrations showing the 
grading for adding a priority; 
Figure 29 is an illustration showing a method for 
assigning a priority to multi-resolution video data; 
Figure 30 is an illustration showing a method for 
constituting a communication payload; 
Figure 31 is an illustration showing a method for 
making data correspond to a communication pay- 
load; 

Figure 32 is an illustration showing the relation 
between object priority, stream priority, and commu- 
nication packet priority; 

Figure 33 is a block diagram of a transmitter of the 
first embodiment of the present invention; 
Figure 34 is an illustration of the first embodiment; 
Figure 35 is a block diagram of the receiver of the 
third embodiment of the present invention; 
Figure 36 is a block diagram of the receiver of the 
fifth embodiment of the present invention; 
Figure 37 is an illustration of the fifth embodiment; 
Figure 38 is a block diagram of the transmitter of the 
sixth embodiment of the present invention; 
Figure 39 is a block diagram of the transmitter of the 
eighth embodiment of the present invention; 
Figure 40 is a flowchart of ttie transmission method 
of the second embodiment of the present invention; 
Figure 41 is a flowchart of the reception method of 
the fourth embodiment of the present invention; 
Figure 42 is a flowchart of the transmission method 
of the seventh embodiment of the present inven- 
tion; 

Figure 43 is a flowchart of the transmission method 
of the ninth embodiment of the present invention; 
Figure 44 is a block diagram showing an audio- 
video transmitter of the present invention; 
Figure 45 is a block diagram showing an audio- 
video receiver of the present invention; 
Figure 46 is an illustration for explaining priority 
adding means for adding a priority to a video and 
audio of an audio-video transmitter of the present 
invention; and 

Figure 47 is an illustration for explaining priority 
deciding means for deciding whether to perform 
decoding by interpreting the priority added to a 
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video and audio of an audio-video receiver of the 
present invention. 

(Description of Symbols) 

5 

[0060] 

11 Reception control section 

12 Separating section 

1 3 Transmitting section 10 

14 Video extending section (Picture extending sec- 
tion) 

15 Video-extension control section (Picture-exten- 
sion control section) 

1 6 Video synthesizing section (Picture synthesizing 15 
section) ^ 

17 Output section 

18 Terminal control section 

401 1 Transmission control section 

4012 Video encoding section (Picture encoding 20 
section) 

4013 Reception control section 

4014 Video decoding section (Picture decoding 
section) 

401 5 Video synthesizing section (Picture synthesiz- 25 
ing section) 

4016 Output section 

4101 Video encoder (Picture encoder) 

41 02 Video decoder (Picture decoder) 

301 Receiving means 30 

302 Estimating means 

303 Video decoder (i.e. Dynamic-picture or Moving 
picture decoder) 

304 Frequency reducing means 

306 Output terminal 35 

307 Input terminal 

3031 Variable decoding means 

3032 Inverse orthogonal transforming means 

3033 Switching unit 

3034 Movement compensating means 40 

3035 Execution-time measuring means 

Best Mode for Carrying Out the Invention 

[0061] Embodiments of the present invention are 4S 
described below by referring to the accompanying draw- 
ings. 

[0062] The embodiments described below mainly 
solve any one of the above problems (A1) to (A6). 
[0063] A "picture (or video)" used tor the present so 
invention includes a static-picture and a moving-picture. 
Moreover, a purposed picture can be a two-dimensional 
picture like computer graphics (CG) or three-dimen- 
sional picture data constituted with a wire-frame model. 
[0064] Figure 1 is a schematic block diagram of the ss 
audio-video transceiver of an embodiment of the 
present invention. 

[0065] In Figure 1 , a reception control section 1 1 for 



receiving information and a transmitting section 13 for 
transmitting information are information transmitting 
means such as a coaxial cable, CATV, LAN, and 
modem. Communication environment can be the envi- 
ronment in which a plurality of logical transmission lines 
can be used without considering multiplexing means 
such as internet or the environment in which multiplex- 
ing means must be considered such as analog tele- 
phone or satellite broadcast 

[0066] Moreover, a system for bidirectionally transfer- 
ring video and audio between terminals such as a pic- 
ture telephone or teleconference system or a system for 
broadcasting broadcast-type video and audio through 
satellite broadcast, CATV, or internet are listed as termi- 
nal connection systems. The present invention takes 
such terminal connection systems into consideration. 
[0067] A separating section 12 shown in Figure 1 is 
means for analyzing received information and separat- 
ing data from control information. Specifically, the sec- 
tion 12 is means for decomposing the header 
information for transmission added to data and data or 
decomposing the header for data control added to the 
data and the contents of the data. A picture extending 
section 14 is means for extending a received video. For 
example, a video to be extended can be the com- 
pressed picture of a standardized moving(dynamic) or 
static picture such as H.261 , H.263, MP EG 1/2, or JPEG 
or not. 

[0068] The picture-extension control section 1 5 shown 
in Figure 1 is means for monitoring the extended state 
of a video. For example, by monitoring the extended 
state of a picture, it is possible to empty-read a receiving 
buffer without extending the picture when the receiving 
buffer almost causes overflow and restart the extension 
of the picture after the picture is ready for extension. 
[0069] Moreover, in Figure 1 , a picture synthesizing 
section 16 is means for synthesizing an extended pic- 
ture. A picture synthesizing method can be defined by 
describing a picture and its structural information (dis- 
play position and cGsplay time (moreover, a display 
period can be included)), a method for grouping pic- 
tures, a picture display layer (depth), an object ID 
(SSRC to be described later), and the relation between 
attributes of them with a script language such as JAVA, 
VRML, or MHEG. The script describing the synthesizing 
method is input or output through a network or a local 
memory. 

[0070] Moreover, an output section 17 is a display or 
printer for outputting a picture synthesized result. A ter- 
minal control section 1 8 is means for controlling each 
section. Furthermore, it is possible to use a structure for 
extending an audio instead of a picture (it is possible to 
constitute the structure by changing a picture extending 
section to an audio extending section, a picture exten- 
sion control section to an audio extension control sec- 
tion, and a picture synthesizing section to an audio 
synthesizing section) or a structure for extending a pic- 
ture and an audio and synthesizing and displaying them 
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while keeping temporal synchronization. 
[0071 ] Furthermore, it is possible to transmit a picture 
and an audio by using a picture compressing section for 
compressing a picture, a picture compression control 
section for controlling the picture compressing section, s 
an audio compressing section for compressing an 
audio, and an audio compression control section for 
controlling the audio compressing section. 
[0072] Figure 2 is an illustration showing a reception 
control section and a separating section. 10 
[0073] By constituting the reception control section 1 1 
shown in Figure 1 with a data receiving section 101 for 
receiving data and a control information receiving sec- 
tion 102 for receiving the control information for control- 
ling data and the separating section 12 with a is 
transmission format storing section 103 for storing a 
transmission structure (to be described later in detail) 
for interpreting transmission contents and a transmis- 
sion information interpreting section 1 04 for interpreting 
transmission contents in accordance with the transmis- 20 
sion structure stored in the transmission format storing 
section 103, it is possible to independently receive data 
and control information. Therefore, for example, it is 
easy to delete or move a received video or audio while 
receiving it. 25 
[0074] As described above, it is possible for the com- 
munication environment purposed by the reception con- 
trol section 11 to use a communication environment 
(internet profile) in which a plurality of logical transmis- 
sion lines can be used without considering multiplexing 30 
means like internet or a communication environment 
(Raw profile) in which multiplexing means must be con- 
sidered like analog telephone or satellite broadcast. 
However, a user premises a communication environ- 
ment in which a plurality of logical transmission lines 35 
(logical channels) are prepared (for example, in the 
case of a communication environment in which TCP/IP 
can be used, the expression referred to as "communica- 
tion port" is generally used). 

[0075] Moreover, as shown in Figure 2, it is assumed 40 
that the reception control section 11 receives one type 
of data transmission line or more and one type of control 
logical transmission line for controlling data to be trans- 
mitted or more. It is also possible to prepare a plurality 
of transmission lines for transmitting data and only one 45 
transmission line for controlling data. Moreover, it is 
possible to prepare a transmission line for controlling 
data every data transmission like the RTP/RTCP also 
used for H.323. Furthermore, when considering the 
broadcast using UDP, it is possible to use a communica- so 
tion system using a single communication port (multi- 
cast address). 

[0076] Figure 3 is an illustration for explaining a 
method for transmitting and controlling video and audio 
by using a plurality of logical transmission lines. The 55 
data to be transmitted is referred to as ES (Elementary 
Stream), which can be picture information for one frame 
or picture information in GOBs or macroblocks smaller 
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than one frame in the case of a picture. 
[0077] In the case of an audio, it is possible to use a 
fixed length decided by a user. Moreover, the data-con- 
trol header information added to the data to be transmit- 
ted is referred to as AL (Adaptation Layer information). 
The information showing whether it is a start position 
capable of processing data, information showing data- 
reproducing time, and information showing the priority 
of data processing are listed as the AL information. Data 
control information of the present invention corresponds 
to the AL information. Moreover, it is not always neces- 
sary for the ES and AL used for the present invention to 
coincide with the contents defined by MPEQ1/2. 
[0078] The information showing whether it is a start 
position capable of processing data specifically includes 
too types of information. First one is a flag for random 
access, that is, the information showing that it can be 
individually read and reproduced independently of pre- 
ceding or following data such as infra-frame (I picture) in 
the case of a picture. Second one is the information 
capable of defining an access flag as a flag for showing 
that it can be individually read, that is, the information 
showing that it is the head of pictures in GOBs or mac- 
roblocks in the case of a picture. Therefore, absence of 
an access flag shows the middle of data Both random 
access flag and access flag are not always necessary 
as the information showing that it is a start position 
capable of processing data. 

[0079] There is a case in which no problem occurs 
even if both the flags are not added in the case of the 
real time communication such as a teleconference sys- 
tem. However, to simply perform edition, a random 
access flag is necessary. It is also possible to decide 
whether a flag is necessary or which flag is necessary 
through a communication channel before transferring 
data. 

[0080] The information indicating a data reproducing 
time shows the information for time synchronization 
when a picture and an audio are reproduced, which is 
referred to as PTS (Presentation Time Stamp) in the 
case of MEPG1/2. Because time synchronisation is not 
normally considered in the case of the real time commu- 
nication such as a teleconference system, the informa- 
tion representing a reproducing time is not always 
necessary. The time interval between encoded frames 
may be necessary information. 
[0081] By making the receiving side adjust a time 
interval, it is possfole to prevent a large fluctuation of 
frame intervals. However, by making the receiving side 
adjust the reproducing interval, a delay may occur. 
Therefore, it may be decided that the time information 
showing the frame interval between encoded frames is 
unnecessary. 

[0082] To decide whether the information showing a 
data reproducing time represents a PTS or frame inter- 
val, it is also possible to decide that the data reproduc- 
ing time is not added to data before transmitting the 
data and communicate the decision to a receiving termi- 
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nal through the communication channel and transmit 
the data together with decided data control information. 
[0083] When the information showing the priority for 
processing data cannot be processed or transmitted 
due to the load of a receiving terminal or that of a net- 
work, it is possible to reduce the load of the receiving 
terminal or network by stopping the processing or trans- 
mission of the data. 

[0084] The receiving terminal is able to process the 
data with the picture-extension control section 15 and 
the network is able to process the data with a relay ter- 
minal or router. The priority can be expressed by a 
numerical value or a flag. Moreover, by transmitting the 
offset value of the information showing the data- 
processing priority as control information or data control 
information (AL. information) together with data and add- 
ing the offset value to the priority previously assigned to 
a video or audio in the case of a sudden fluctuation of 
the load of a receiving terminal or network, it is possible 
to set a dynamic priority corresponding to the operation 
state of a system. 

[0085] Furthermore, by transmitting the information for 
identifying presence/absence of scramble, pres- 
ence/absence of copyright, and original or copy as con- 
trol information together with a data identifier (SSRC) 
separately from data as control information, it is simpli- 
fied to cancel the scramble at a relay node. 
[0086] Moreover, the information showing the data 
processing priority can be added every stream consti- 
tuted with the aggregation of frames of a plurality of pic- 
tures or audios or every frame of video or audio. 
[0087] Priority adding means for deciding the 
encoded-information processing priority under overbad 
in accordance with the predetermined rules by the 
encoding method such as H.263 or G.723 and making 
the encoded information correspond to the decided pri- 
ority is provided for a transmitting terminal unit (see Fig- 
ure 46). 

[0088] Figure 46 is an illustration for explaining priority 
adding means 5201 for adding a priority to a picture and 
an audio. 

[0089] That is, as shown in Figure 46, a priority is 
added to encoded-video data (to be processed by video 
encoding means 5202) and encoded-audio data (to be 
processed by audio encoding means 5203) in accord- 
ance with predetermined rules. The rules for adding pri- 
orities are stored in priority adding rules 5204. The rules 
include rules for adding a priority higher than that of a P- 
f rame (inter-frame encoded picture frame) to an l-frame 
(intra -frame encoded picture frame) and rules for adding 
a priority lower than that of an audio to a picture. More- 
over, it is possible to change the rules in accordance 
with the designation of a user. 
[0090] Priority-adding objects are scene changes in 
the case of a picture or an audio block and audioless 
block in the case of a picture frame, stream, or audio 
designated by an editor or user. 
[0091] To add a priority in picture or audio frames for 



defining the processing priority under overload, the fol- 
lowing methods are considered: a method for adding a 
priority to a communication header and a method for 
embedding a priority in the header of a bit stream in 

5 which a video or audio is encoded under encoding. The 
former makes it posstole to obtain the information for 
priority without decoding it and the latter makes it possi- 
ble to independently handle a single bit stream without 
depending on a system. 

w [0092] When one picture frame (e.g. intra-frame 
encoded l-frame or inter-frame encoded P- or B-f rame) 
is divided into a plurality of transmission packets, a pri- 
ority is added only to a communication header for trans- 
mitting the head of a picture frame accessible as 

15 independent information in the case of a picture (when 
priorities are equal in the same picture frame, it is pos- 
sible to assume that the priorities are not changed 
before the head of the next accessfole picture frame 
appears). 

20 [0093] Moreover, it is possible to realize configuration 
in accordance with control information by making the 
range of a value capable of expressing a priority varia- 
ble (for example, expressing time information with 16 
bits or 32 bits depending on the purpose). 

25 [0094] Furthermore, in the case of a decoder, priority 
deciding means for deciding a processing method is 
provided for a receiving terminal unit in accordance with 
the priority under overload of received various encoded 
pieces of information (see Figure 47). 

30 [0095] Figure 47 is an illustration for interpreting Prior- 
ities added to a picture and an audio and explaining pri- 
ority deciding means 5301 for deciding whether to 
perform decoding. 

[0096] That is, as shown in Figure 47, the priorities 
35 include a priority added to each stream of each picture 
or audio and a priority added to each frame of a picture 
or audio. It is possible to use these priorities independ- 
ently or by making a frame priority correspond to a 
stream priority. The priority" deciding means 5301 
40 decides a stream or frame to be decoded in accordance 
with these priorities. 

[0097] Decoding is performed by using two types of 
priorities for deciding a processing priority under over- 
load at a terminal. 

45 [0098] That is, a stream priority (inter-time-series pri- 
ority) for defining a relative priority between bit streams 
such as a picture and audio and a frame priority (intra- 
time-series priority) for defining a relative priority 
between decoding units such as picture frames in the 

so same stream are defined (Figure 24). 

[0099] The former stream priority makes it possible to 
handle a plurality of videos or audios. The latter frame 
priority makes it possible to change scenes or add dif- 
ferent priorities even to the same intra-frame encoded 

55 picture frames (l-frame) in accordance with the intention 
of an editor. 

[0100] By making a stream priority correspond to a 
time assigned to an operating system (OS) for encoding 
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or decoding a picture or audio or a processing priority 
and thereby controlling the stream priority, it is possible 
to control a processing time at an OS level. For exam- 
ple, in the case of Windows95/NT of Microsoft Corpora- 
tion, a priority can be defined at five OS levels. By 5 
realizing encoding or decoding means by software in 
threads, it is possible to decide a priority at an OS level 
to be assigned to each thread in accordance with the 
stream priority of a purposed stream. 
[0101] The frame priority and stream priority 10 
described above can be applied to a transmission 
medium or data-recording medium. For example, by 
defining the priority of a packet to be transmitted as an 
access unit priority, it is possbie to decide a priority con- 
cerned with packet transmission or a priority for 75 
processing by a terminal under overload in accordance 
with the relation between frame priority and stream pri- 
ority such as the relation of Access Unit Priority = 
Stream Priority - Frame Priority. 

[0102] Moreover, it is possible to decide a priority by 20 
using a floppy disk or optica! disk as a data-recording 
medium. Furthermore, it is possible to decide a priority 
by using not only a recording medium but also an object 
capable of recording a program such as an IC card or 
ROM cassette. Furthermore, it is possible to use a 25 
repeater for a picture or audio such as a router or gate- 
way for relaying data. 

[01 03] As a specific method for using a priority, when 
a receiving terminal is overloaded, priority deciding 
means for deciding the threshold of the priority of 30 
encoded information to be processed is set to a picture- 
extension control section or audio-extension control 
section and the time to be displayed (PTS) is compared 
with the elapsed time after start of processing or the 
time to be decoded (DTS) is compared with the time 35 
elapsed time after start of processing to change thresh- 
olds of the priority of encoded information to be proc- 
essed in accordance with the comparison result (it is 
also possible to refer to the insertion interval of i-frame 
or the grading of a priority as the information for chang- 40 
ing thresholds). 

[0104] in the case of the example shown in Figure 
20(a), a picture with the size of captured QCIF or CIF is 
encoded by an encoder (H.263) under encoding to out- 
put a time stamp (PTS) showing the time for decoding 45 
(DTS) or the time for displaying the picture, priority infor- 
mation showing processing sequence under overload 
(CGD. Computational Graceful Degradation), frame 
type (SN). and sequence number together with 
encoded information. so 
[0105] Moreover, in the case of the example shown in 
Figure 20(b), an audio is also recorded through a micro- 
phone and encoded by an encoder (G.721) to output a 
time stamp (PTS) showing the time for decoding (DTS) 
or the time for reproducing an audio, priority information 55 
(CGD), and sequence number (SN) together with 
encoded information. 

[0106] Under decoding, as shown in Figure 20(c). a 



picture and an audio are supplied to separate buffers to 
compare their respective DTS (decoding time) with the 
elapsed time after start of processing. When DTS is not 
delayed, the picture and the audio are supplied to their 
corresponding decoders (H.263 and G.721). 
[01 07] The example in Figure 21 describes a method 
for adding a priority by an encoder under overload. For 
a picture, high priorities of "0" and "1 " are assigned to I- 
frame (intra-frame encoded picture frame) (the smaller 
a numerical becomes, the lower a priority becomes). P- 
frame has a priority of "2" which is lower than that of I- 
frame. Because two levels of priorities are assigned to I- 
frame, it is possible to reproduce only l-frame having a 
priority of "0" when a terminal for decoding has a large 
load. Moreover, it is necessary to adjust the insertion 
interval of l-frame in accordance with a priority adding 
method. 

[01 08] The example in Figure 22 shows an illustration 
showing a method for deciding a priority at a receiving 
terminal under overload. The priority of a frame to be 
disused is set to a value larger than a cutOffPriority. 
That is, every picture frame is assumed as an object to 
be processed. It is possible to previously know the max- 
imum value of priorities added to picture frames by com- 
municating it from the transmitting side to the receiving 
side (step 101). 

[01 09] When DTS is compared with the elapsed time 
after start of processing and resultantly, the elapsed 
time is larger than DTS (when decoding is not in time), 
the threshold of the priority of a picture or audio to be 
processed is decreased to thin out processings (step 
102). However, when the elapsed time after start of 
processing is smaller than DTS (decoding is in time), 
the threshold of a priority is increased in order to 
increase the number of pictures or audio which can be 
processed (step 103). 

[01 1 0] If the image from one before is skipped by P- 
frame, no processing is performed. If not, a priority off- 
set value is added to the priority of a picture frame (or 
audio frame) to compare the priority offset value with 
the threshold of the priority. When the offset value does 
not exceed the threshold, data to be decoded is sup- 
plied to a decoder (step 1 04). 
[01 1 1 ] A priority offset allows the usage of previously 
checking the performance of a machine and communi- 
cating the offset to a receiving terminal (it is also possi- 
ble that a user issues designation at the receiving 
terminal) and the usage of changing priorities of a plu- 
rality of video and audio streams in streams (for exam- 
ple, thinning out processings by increasing the offset 
value of the rearmost background). 
[0112] When a multi-stream is purposed, it is also 
possible to add a priority for each stream and decide the 
skip of decoding of a picture or audio. Moreover, in the 
case of real time communication, it is possible to decide 
whether decoding is advanced or delayed at the termi- 
nal by handling the TR (Temporary Reference) of H.263 
similarly to DTS and realize the skipping same as 
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described above. 

[0113] Figure 23 is an illustration showing temporal 
change of priorities by using the above-algorithm. 
[0114] Figure 23 shows the change of a priority to be 
added to a picture frame. This priority is a priority for 5 
deciding whether to perform decoding when a terminal 
is overloaded, which is added every frame. The smaller 
the value of a priority becomes, the higher the priority 
becomes. In the case of the example in Figure 23, 0 has 
the highest priority. When the threshold of a priority is 3, 10 
a frame having a priority to which a value larger than 3 
is added is disused without being decoded and a frame 
having a priority to which a value of 3 or less is added is 
decoded. By selectively discussing frames in accord- 
ance with priorities, it is possible to control the load of a 75 
terminal. It is also possible to dynamically decide the 
priority threshold in accordance with the relation 
between the present processing time and the decoding 
time (DTS) to be added to each frame. This technique 
can be applied not only to a picture frame but also to an 20 
audio in accordance with the same procedure. 
[011 51 In the case of a transmission line such as inter- 
net, when it is necessary to retransmit encoded infor- 
mation lost under transmission, it is possible to 
retransmit only a picture or audio required by the recerv- 2s 
ing side by providing a retransmission request priority 
deciding section for deciding the threshold of the priority 
of the encoded information to be retransmitted for a 
reception control section and deciding the threshold of 
the priority added to the encoded information whose 30 
retransmission should be requested in accordance with 
the information for priority, retransmission frequency, 
loss rate of information, insertion interval of intra-frame 
encoded frame, grading of priority (e.g. five-level prior- 
ity) which are controlled by the priority deciding section. 35 
If the retransmission frequency or loss rate of informa- 
tion is too large, it is necessary to raise the priority of the 
information to be retransmitted and lower the retrans- 
mission or loss rate. Moreover, by Knowing the priority 
used for the priority deciding section, it is possible to 40 
prevent the information to be processed from being 
transmitted. 

[01 1 6] In the case of a transmitting terminal, when an 
actual transfer rate exceeds the target transfer rate of 
the information of the transmitting terminal or when writ- 45 
ing of the encoded information into a transmitting buffer 
is delayed as the result of comparing the elapsed time 
after start of transfer processing with the time added to 
the encoded information to be decoded or displayed, it 
is possible to transmit a picture or audio matching with so 
the target rate by using a priority added to encoded 
information and used by the priority deciding section of 
the receiving terminal when the terminal is overloaded 
and thereby thinning out transmissions of information. 
Moreover, by introducing the processing skipping func- ss 
tion under overload performed at the receiving-side ter- 
minal into the transmitting-side terminal, it is possible to 
control a failure due to overload of the transmitting-side 



terminal. 

[0117] By making it possible to transmit only neces- 
sary information out of the above-described AL informa- 
tion according to necessity, ft is possible to adjust the 
amount of Information to be transmitted to a narrow- 
band communication channel such as an analog tele- 
phone line. It is possible to recombine the AL informa- 
tion (data control information) used for the transmitting 
side by deciding the data control information to be 
added to data at a transmitting-side terminal before 
transmitting the data, communicating the data control 
information to be used to a receiving terminal as control 
information (for example, using only a random access 
flag), and rewriting at the receiving-side terminal based 
on the obtained control information the information 
about a transmission structure (showing which AL infor- 
mation is used) stored in the transmission format storing 
section 103 (see Figure 16). 

[0118] Figure 4 is an illustration for explaining a 
method for dynamically changing header information 
added to the data for a picture or audio to be transmit- 
ted. In the case of the example in Figure 4, the data 
(ES) to be transmitted is decomposed into data pieces 
and the identifying information (sequence number) for 
showing the sequence of data, the information (marker 
bit) showing whether it is a start position capable of 
processing data pieces, and time information (time 
stamp) concerned with transfer of data pieces are 
added to data pieces in the form of communication 
headers by assuming that the above pieces of informa- 
tion correspond to transmission control information of 
the present invention. 

[0119] Specifically, RTP (Realtime Transfer Protocol, 
REC1889) uses the information for the above sequence 
number, marker bit, time stamp, object ID (referred to as 
SSRC), and version number as communication head- 
ers. Though a header-information item can be 
extended, the above items are always added as fixed 
items. However, when the realtime communication such 
as the case of a video telephone and transmission of 
accumulated media such as the case of video-on- 
demand are present together in an environment in 
which a plurality of different encoded pictures or audio 
are simultaneously transmitted, identifying means is 
necessary because meanings of communication head- 
ers are different from each other. 
[0120] For example, time-stamp information shows 
PTS that is a reproducing time as previously descrfoed 
in the case of MPEG1/2. In the case of H.261 or H.263, 
however, the time-stamp information shows a time inter- 
val when the information is encoded. However, to proc- 
ess H.263 synchronously with an audio, it is necessary 
to show that a time stamp is PTS information. This is 
because time-stamp information shows the time interval 
between encoded frames in the case of. H. 263 and it is 
defined by RTP that the time stamp of the first frame is 
random. 

[01 21 ] Therefore, it is necessary to add a flag showing 
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whether a time stamp is PTS as (a) communication 
header information (it is necessary to extend a commu- 
nication header) or (b) header information for payload of 
H.263 or H.261 (that is, AL information) (in this case, it 
is necessary to extend payload information). 5 
[01 22] A marker bit serving as the information show- 
ing whether it is a start position capable of processing 
data pieces is added as RTP header information. More- 
over, as described above, there is a case in which it is 
necessary to provide an access flag showing that it is a 10 
start position capable of accessing data and a random 
access flag showing that it is possible to access data at 
random for AL information. Because doubly providing 
flags for a communication header lowers the efficiency, 
a method of substituting an AL flag by a flag prepared is 
for the communication header is also considered. 

. (c) The problem is solved by newly providing a flag 
showing that an AL flag is substituted by the header 
added to a communication header without adding a 20 
flag to AL for the communication header or defining 
that the marker bit of the communication header is 
the same as that of AL (it is expected that interpre- 
tation can be quickly performed compared to the 
case of providing a flag for AL). That is, a flag is 25 
used which shows whether the marker bit has the 
same meaning as the flag of AL. In this case, it is 
considered to improve the communication header 
or describe it in an extension region. 

30 

[01 23] However, (d) it is also possible to interpret the 
meaning of the marker bit of the communication header 
so as to mean that at least either of a random access 
flag and an access flag is present in AL. In this case, it 
is possible to knew that the meaning of interpretation is 35 
changed from the conventional case by the version 
number of the communication header. Moreover, 
processing is simplified by providing an access flag or 
random access flag only for the communication header 
or the header of AL (for the former, a case of providing 40 
the flag for both the headers is considered but it is nec- 
essary to newly extend the communication header). 
[0124] It is already described to add the information 
showing the priority of data processing as the informa- 
tion for AL By adding the data-processing priority to the 45 
communication header, it is possible to decide the 
processing of the data-processing priority without inter- 
preting the contents of data also on a network. Moreo- 
ver, in the case of IPv6, it is possible to add the priority 
at a layer lower than the level of RTP. so 
[0125] By adding a timer or counter for showing the 
effective period of data processing to the communica- 
tion header of RTP, it is possible to decide how the state 
of a transmitted packet changes. For example, when 
necessary decoder software is stored in a memory hav- ss 
ing a low access speed, it is possible to decide the infor- 
mation required by a decoder and when the information 
is required by a timer or counter. In this case, the infor- 



mation for the priority of a timer or counter or the infor- 
mation for the priority of data processing is unnecessary 
for AL information depending on the purpose. 
[01 26] Figures 5(a) and 5(b) and Figures 6(a) to 6(d) 
are illustrations for explaining a method for adding AL 
information. 

[01 27] By sending the control information for commu- 
nicating whether to add AL to only the head of the data 
to be transmitted as shown in Figure 5(a) or whether to 
add AL to each data piece after decomposing the data 
to be transmitted (ES) into one data piece or more to a 
receiving terminal as shown in Figure 5(b), it is possible 
to select the grading for handling transmission informa- 
tion. Adding AL to subdivided data is effective when 
access delay is a problem. 

[0128] As described above, to previously communi- 
cate recombination of data control information at the 
receiving side or change of methods for arranging data 
control information to data to a receiving-side terminal, 
receiving-terminal correspondence can be smoothly 
performed by using the expression of a flag, counter, or 
timer and thereby, preparing the expression as AL infor- 
mation or as a communication header to communicate 
it to the receiving terminal. 

[0129] In the case of the above examples, a method 
for avoiding duplication of the header of RTP (or com- 
munication header) with AL information and a method 
for extending the communication header of RTP or AL 
information are described. However, it is not always 
necessary for the present invention to use RTF. For 
example, it is possible to newly define an original com- 
munication header or AL information by using UDP or 
TCP Though the internet profile uses RTP sometimes, 
a multifunctional header such as RTF is not defined in 
the Raw profile. The following four types of concepts are 
considered for AL information and communication 
header (see Figures 6(a) to 6(d)). 

(1) The header information of RTF or AL informa- 
tion is corrected and extended so that the header 
information already assigned to RTP and that 
already assigned to AL are not overlapped (particu- 
larly, the information for a time stamp is overlapped 
and the priority information for a timer, counter, or 
data processing becomes extension information). 
Or, it is possible to use a method of not extending 
the header of RTP or not considering duplication of 
AL information with information of RTP. They corre- 
spond to the contents having been shown so far. 
Because a part of RTP is already practically used 
for H.323, it is effective to extend RTP having com- 
patibility. (See Figure 6(a).) 

(2) Independently of RTP, a communication header 
is simplified (for example, using only a sequence 
number) and remainder is provided for AL informa- 
tion as multifunctional control information. Moreo- 
ver, by making it possible to variably set items used 
for AL information before communication, it is pos- 
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sible to specify a flexible transmission format. (See 
Figure 6(b).) 

(3) Independently of RTP, AL information is simpli- 
fied (for an extreme example, no information is 
added to AL) and every control information is pro- 
vided for a communication header. A sequence 
number, time stamp, marker bit, payload type, and 
object ID frequently used as communication head- 
ers are kept as fixed information and data-process- 
ing priority information and timer information are 
respectively provided with an identifier showing 
whether extended information is present as 
extended information to refer to the extended infor- 
mation if the information is defined. (See Figure 
6(c).) 

(4) Independently of RTP, a communication header 
and AL information are simplified and a format is 
defined as a packet separate from the communica- 
tion header or AL information to transmit the format 
For example, a method is also considered in which 
only a marker bit, time stamp, and object ID are 
defined for AL information, only a sequence 
number is defined for a communication header, and 
payload information, data-processing priority infor- 
mation, and timer information are defined as a 
transmission packet (second packet) separate from 
the above information and transmitted. (See Figure 
6(d).) 

[0130] As described above, when considering a pur- 
pose and header information already added to a picture 
or audio, it is preferable so as to be able to freely define 
(customize) a packet (second packet) to be transmitted 
separately from a communication header, AL informa- 
tion, or data in accordance with the purpose. 
[0131] Figure 7 is an illustration for explaining a 
method for transmitting information by dynamically mul- 
tiplexing and separating a plurality of logical transmis- 
sion lines. The number of logical transmission lines can 
be decreased by providing an information multiplexing 
section capable of starting or ending multiplexing of the 
information for logical transmission lines for transmitting 
a plurality of pieces of data or control information in 
accordance with the designation by a user or the 
number of logical transmission lines for a transmitting 
section and an information separating section for sepa- 
rating multiplexed information for a reception control 
section. 

[0132] In Figure 7, the information multiplexing section 
is referred to as "Group MUX" and specifically, it is pos- 
sible to use a multiplexing system such as H.223. it is 
possible to provide the Group MUX for a transmit- 
ting/receiving terminal. By providing the Group MUX for 
a relay router or terminal, it is possible to correspond to 
a narrow-band communication channel. Moreover, by 
realizing Group MUX with H.223, it is possible to inter- 
connect H.223 and H.324. 

[0133] To quickly fetch the control information (multi- 



plexing control information) for the information multi- 
plexing section, it is possible to reduce a delay due to 
multiplexing by transmitting the control information in 
the information multiplexing section through another 

5 logical transmission line without multiplexing the control 
information with data by the information multiplexing 
section. Thereby, it is possible for a user to select 
whether to keep the consistency with conventional mul- 
tiplexing or reduce a delay due to multiplexing by corn- 
to municating and transmitting whether to multiplex the 
control information concerned with the information mul- 
tiplexing section with data and transmit them or transmit 
the control information through another logical transmis- 
sion line without multiplexing the information with the 

15 data. In this case, the multiplexing control information 
concerned with the information multiplexing section is 
information showing the content of multiplexing about 
how the information multiplexing section performs multi- 
plexing for each piece of data. 

zo [0134] As described above, similarly, it is possible to 
transmit the notification of a method for transmitting at 
least the information for communicating the start and 
end of multiplexing, information for communicating the 
combination of logical transmission lines to be mutti- 

25 plexed, and control information concerned with multi- 
plexing (multiplexing control information) as control 
information in accordance with an expression method 
such as a flag, counter, or timer or reduce the setup time 
at the receiving side by transmitting data control infbr- 

30 mation to a receiving-side terminal together with data. 
Moreover, as previously described, it is possible to pro- 
vide an item for expressing a flag, counter, or timer for 
the transmission header of RTP. 
[0135] When a plurality of information multiplexing 

35 sections or a plurality of information separating sections 
are present, it is possible to identify to which information 
multiplexing section the control information (multiplex- 
ing control information) belongs by transmitting the con- 
trol information (multiplexing control information) 

40 together with an identifier for identifying an information 
multiplexing section or information separating section. 
The control information (multiplexing control informa- 
tion) includes a multiplexing pattern. Moreover, by using 
a table of random number and thereby, deciding an 

45 identifier of an information multiplexing section or infor- 
mation separating section between terminals, it is pos- 
sible to generate an identifier of the information 
multiplexing section. For example, it is possible to gen- 
erate random numbers in a range determined between 

so transmitting and receiving terminals and use the largest 
value for the identifier (identification number) of the 
information multiplexing section. 
[0136] Because the data multiplexed by the informa- 
tion multiplexing section is conventionally different from 

55 the media type defined in RTP, it is necessary to define 
the information showing that it is information multiplexed 
by the information multiplexing section (new media type 
H.223 is defined) for the payload type of RTP. 
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[01 37] By arranging the information to be transmitted 
by or recorded in the information multiplexing section in 
the sequence of control information and data informa- 
tion so as to improve the access speed to multiplexed 
data, it is expected to quickly analyze multiplexed infor- 5 
mation. Moreover, it is possible to quickly analyze 
header information by fixing an item which is described 
in accordance with the data control information added to 
control information and adding and multiplexing an 
identifier (unique pattern) different from data. 10 
[0138] Figure 8 is an illustration for explaining the 
transmission procedure of a broadcasting program. By 
using the relation between the identifier of a logical 
transmission line and the identifier of a broadcasting 
program as the information of the broadcasting program 75 
and thereby, transmitting control information or adding 
the identifier of a broadcasting program to data as data 
control information (AL information), it is possible to 
identify that the data transmitted through a plurality of 
transmission lines is broadcasted for which program. 20 
Moreover, by transmitting the relation between the iden- 
tifier of data (SSRC in the case of RTP) and the identi- 
fier of a logical transmission line (e.g. port number of 
LAN) to a receiving-side terminal as control information 
and transmitting corresponding data after it is confirmed 25 
that the control information can be received by the 
receiving-side terminal (Ack/Reject), it is possible to 
form the correspondence between data pieces even if 
control information and data are respectively transmit- 
ted through an independent transmission line. 30 
[0139] By combining an identifier showing the trans- 
mission sequence of broadcasting programs or data 
pieces with the information for a counter or timer for 
showing a term of validity in which broadcasting pro- 
gram or data can be used as information, adding the 35 
combined identifier and information to the broadcasting 
program or data, and transmitting them, it is possible to 
realize broadcasting without return channel (when the 
term of validity almost expires, reproduction of the infor- 
mation or data for a broadcasting program is started 40 
even if information is insufficient). Moreover, a method 
can be considered in which control information and data 
are broadcasted without being separated from each 
other by using the address of a single communication 
port (multicast address). 45 
[0140] In the case of communication with no back 
channel, it is necessary to transmit control information 
sufficiently before transmitting data so as to enable the 
receiving terminal to know a structural information of 
data. Moreover, control information should be transmit- so 
ted through a transmission channel free from packet 
loss and having a high reliability. However, when using a 
transmission channel having a low reliability, it is neces- 
sary to cyclically transmit the control information having 
the same transmission sequence number. This is not ss 
restricted to the case of transmitting the control informa- 
tion concerned with a setup time. 
[0141] Moreover, it is possible to flexibly control and 
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transmit data by selecting an item which can be added 
as data control information (e.g. access flag, random 
access flag, data reproducing time (PTS), or data- 
processing-priority information), deciding whether to 
transmit the data control information together with the 
identifier (SSRC) of data as control information through 
a logical transmission line different from that of the data 
or transmit the data control information as data control 
information (information for AL) together with the data at 
the transmitting side before transmitting the data, and 
communicating and transmitting the data to the receiv- 
ing side as control information. 
[01 42] Thereby, it is possible to transmit data informa- 
tion without adding information to AL. Therefore, to 
transmit the data for a picture or audio by using RTP, it 
is unnecessary to extend the definition of the payload 
having been defined so far. 

[0143] Figures 9(a) and 9(b) are illustrations showing 
a picture or audio transmission method considering the 
read time and rise time of program or data Particularly, 
when the resources of a terminal are limited like the 
case of satellite broadcasting or a portable terminal 
having no return channel and being unidirectional, pro- 
gram or data is present and used at a receiving-side ter- 
minal, a necessary program (eg. H.263, MPEG1/2, or 
software of audio decoder) or data (e.g. video data or 
audio data) is present in a memory (e.g. DVD, hard disk, 
or file server on network) requiring a lot of read time, it 
is possible to reduce the setup time of program or data 
required in advance by previously receiving it as control 
information or receiving it together with data as data 
control information in accordance with the expression 
method such as the identifier for identifying the program 
or data, identifier (e.g. SSRC, or Logical Channel 
Number) of a stream to be transmitted, or a flag, counter 
(count-up/down), or timer for estimating the point of time 
necessary for a receiving terminal (Figure 18). 
[01 44] When program or data is transmitted, by trans- 
mitting the program or dafa from the transmitting side 
together with the information showing the storage desti- 
nation (e.g. hard disk or memory) of the program or data 
at a receiving terminal, time required for start or read, 
relation between the type or storage destination of a ter- 
minal and the time required for start or read (e.g. rela- 
tion between CPU power, storage device, and average 
response time), and utilization sequence, it is possible 
to schedule the storage destination and read time of the 
program or data if the program or data necessary for the 
receiving terminal is actually required. 
[0145] Figures 10(a) and 10(b) are illustrations for 
explaining a method for corresponding to zapping 
(channel change of TV). 

[0146] When it is necessary to execute a program at 
a receiving terminal differently from the case of conven- 
tional satellite broadcasting for receiving only pictures, 
the setup time until the program is read and started is a 
large problem. The same is true for the case in which 
available resources are limited like the case of a porta- 
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ble terminal. 

[01 47] It is expected that the setup time at a receiving- 
side terminal can be decreased by (a) using a main 
iooking-listening section by which the user looks at and 
listens to. and an auxiliary Iooking-listening section In 
which a receiving terminal cyclically monitors programs 
other than the program looked and listened by a user 
and receiving the relation between identifier for identify- 
ing program or data required in advance, information for 
a flag, counter, or timer for estimating the point of time 
necessary for the receiving terminal, and program as 
control information (information transmitted by a packet 
different from that of data to control terminal processing) 
or as data control information (information for AL), and 
preparing read of the program or data together with data 
as one of the settlement measures when to program or 
data necessary for a program other than the program 
looked and listened by the user is present in a memory 
requiring a lot of time for read. 
[0148] It is possible to prevent a screen from stopping 
under setup by setting a broadcasting channel for 
broadcasting only heading pictures of the pictures 
broadcasted through a plurality of channels and switch- 
ing programs by a user, and thereby, when necessary 
program or data is present in a memory requiring a lot 
of time for read, temporarily selecting the heading pic- 
ture of a program required by the user and showing it for 
the user or showing that program or data is currently 
read, and restarting the program required by the user 
after necessary program or data is read by the memory 
as the second one of the settlement measures. The 
above heading pictures include broadcasted pictures 
obtained by cyclically sampling programs broadcasted 
through a plurality of channels. 
[0149] Moreover, a timer is a time expression and 
shows the point of time when a program necessary to 
decode a data stream sent from the transmitting side is 
necessary. A counter is the basic time unit determined 
between transmitting and receiving terminals, which 
can be information showing what-th time. A flag is trans- 
mitted and communicated together with the data trans- 
mitted before the time necessary for setup or control 
information (information transmitted through a packet 
different from that of data to control terminal process- 
ing). It is possible to transmit the timer and counter by 
embedding them in data or transmit them as control 
information. 

[0150] Furthermore, to decide a setup time, the time 
in which setup is performed can be estimated by, when 
using a transmission line such as ISDN operating on the 
clock base, using a transmission serial number for iden- 
tifying a transmission sequence as transmission control 
information in order to communicate from the transmit- 
ting terminal to the receiving terminal a time point when 
program or data is required and thereby communicating 
the serial number to a receiving terminal together with 
data as data control information or as control informa- 
tion. Furthermore, when a transmission time is fluctu- 
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ated due to jitter or delay like internet, it is necessary to 
add the transmission time to the setup time by consider- 
ing the propagation delay of transmission in accordance 
with jitter or delay time by the means for realizing RTCP 
5 (media transmission protocol used for internet). 

[01 51 ] Figures 11 (a) to 1 9(b) are illustrations showing 
specific examples of protocols actually transferred 
between terminals. 

[0152] A transmission format and a transmission pro- 

w cedure are described in ASN.1 . Moreover, the transmis- 
sion format is extended on the basis of H.245 of ITU. As 
shown in Figure 11(a), objects of a picture and audio 
can have a hierarchical structure. In the case of this 
example, each object ID has the attributes of a broad- 

J5 casting-program identifier (program ID) and an object ID 
(S SRC) and the structural information and synthesizing 
method between pictures are described by a script lan- 
guage such as Java or VRML 
[0153] Figure 11(a) is an illustration showing exam- 

20 pies of the relation between objects. 

[0154] In Figure 11(a), objects are media such as an 
audio-video, CG, and text. In the case of the examples 
in Figure 11(a), objects constitute a hierarchical struc- 
ture. Each object has a program number "Program ID" 

25 corresponding to TV channel) and an object identifier 
"Object ID" for identifying an object. When transmitting 
each object in accordance with RTP (media transmis- 
sion protocol for transmitting media used for internet, 
Realtime Transfer Protocol), it is possible to easily iden- 

30 tify the object by making the object identifier correspond 
to SSRC (synchronous source identifier). Moreover, it is 
possible to describe the structure between objects with 
a description language such as JAVA or VRML 
[0155] Two types of methods for transmitting the 

35 objects are considered. One is the broadcasting type in 
which the objects are unilaterally transmitted from a 
transmitting-side terminal. The other is the type (com- 
munication type) for transferring the objects between 
transmitting and receiving terminals (terminals A and 

40 B). 

[0156] For example, it is possible to use RTP as a 
transmission method in the case of internet. Control 
information is transmitted by using a transmission chan- 
nel referred to as LCNO in the case of the standard for 

45 video telephones. In the case of the example in Figure 
1 1(a), a plurality of transmission channels are used for 
transmission. The same program channel (program ID) 
is assigned to these channels. 
[01 57] Figure 1 1 (b) is an illustration for explaining how 

so to realize a protocol for realizing the functions described 
for the present invention. The transmission protocol 
(H.245) used for the video-telephone standards (H.324 
and H.323) is described below. The functions described 
for the present invention are realized by extending 

55 H.245. 

[01 58] The description method shown by the example 
in Figure 11(b) is the protocol description method 
referred to as ASN.1. "Terminal Capability Set" 
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expresses the performance of a terminal. In the case of 
the example In Figure 11(b), the function described as 
n mpeg4 Capability" is extended for the conventional 
H.245. 

[01 59] In Figure 12, "mpeg4 Capability" describes the s 
maximum number of pictures "Max Number Of pictures" 
and the maximum number of audio ("Max Number Of 
Audio") which can be simultaneously processed by a 
terminal and the maximum number of multiplexing func- 
tions ("Max Number Of Mux") which can be realized by io 
a terminal. 

[01S0] In Figure 1 2, these are expressed as the max- 
imum number of objects ("Number Of Process Object") 
which can be processed. Moreover, a flag showing 
whether a communication header (expressed as AL in 15 
Figure 12) can be changed is described. When the 
value of the flag is true, the communication header can 
be changed. To communicate the number of objects 
which can be processed between terminals to each 
other by using "MPEG4 Capability", the communicated 20 
side returns "MEPG4 Capability Ack" to a terminal from 
which "MEPG4 Capability" is transmitted if the commu- 
nicated side can accept (process) the objects but 
returns "MEPG4 Capability Reject" to the terminal if not. 
[01 61 ] Figure 1 3(a) shows how to describe a protocol 25 
for using the above Group MUX for multiplexing a plural- 
ity of logical channels to one transmission channel 
(transmission channel of LAN in the case of this exam- 
ple) in order to share the transmission channel by logi- 
cal channels. In the case of the example in Figure 13(a), 30 
multiplexing means (Group MUX) is made to corre- 
spond to the transmission channel ("LAN Port Number") 
of LAN (Local Area Network). "Group Mux ID" is an 
identifier for identifying the multiplexing means. To 
share the multiplexing means by terminals by using 35 
"Create Group Mux" and perform communication 
between the terminals, the communicated side returns 
"Create Group Mux Ack" to a terminal from which "Cre- 
ate Group Mux" is transmitted if the side can accept 
(use) the multiplexing means but returns "Create Group 40 
Mux Reject" to the terminal if not. Separating means 
serving as means for performing an operation reverse to 
that of the multiplexing means can be realized by the 
same method. 

[01 62] In Figure 1 3(b), a case of deleting already-gen- as 
erated multiplexing means is described. 
[0163] In Figure 13(c), the relation between the trans- 
mission channel of LAN and a plurality of logical chan- 
nels is described. 

[01 64] The transmission channel of LAN is described so 
in accordance with "LAN Port Number" and the logical 
channels are described in accordance with "Logical 
Port Number". 

[0165] In the case of the examples in Figure 13(c), it 
is possible to make the transmission channel of one 55 
IAU correspond to up to 15 logical channels. 
[0166] In Figure 13. when the number of MUXs that 
can be used is only one. Group Mux ID is unnecessary. 



Moreover, to use a plurality of Muxes. Group Mux ID is 
necessary for each command of H.223. Furthermore, it 
is possible to use a flag for communicating the relation 
between ports used between the multiplexing means 
and separating means. Furthermore, it is possible to 
use a command making it possible to select whether to 
multiplex control information or transmit the information 
through another logical transmission line. 
[01 67] In the case of the explanation in Figures 1 3(a) 
to 13(c), the transmission channel uses LAN. However, 
it is also possible to use a system using no internet pro- 
tocol like H.223 or MPEG2. 

[0168] In Figure 14, "Open Logical Channel" shows 
the protocol description for defining the attribute of a 
transmission channel. In the case of the example in Fig- 
ure 14. "MPEG4 Logical Channel Parameters" is 
extended and defined for the protocol of H.245. 
[01 69] Figure 15 shows that a program number (cor- 
responding to a TV channel) and a program name are 
made to correspond to the transmission channel of LAN 
("MPEG4 Logical Channel Parameters"). 
[0170] Moreover, in Figure 15, "Broadcast Channel 
Program" denotes a description method for transmitting 
the correspondence between LAN transmission chan- 
nel and program number in accordance with the broad- 
casting type. The example in Figure 15 makes it 
possible to transmit the correspondence between up to 
1,023 transmission channels and program numbers. 
Because transmission is unilaterally performed from the 
transmitting side to the receiving side in the case of 
broadcasting, it is necessary to cyclically transmit these 
pieces of information by considering the loss during 
transmission. 

[01 71 ] In Figure 1 6(a), the attribute of an object (e.g. 
picture or audio) to be transmitted as a program is 
described ("MPEG4 Object Classdefinition"). Object 
information ("Object Structure Element") is made to cor- 
respond to a program identifier ("Program ID"). It is pos- 
sible to make up to 1,023 objects correspond to 
program identifiers. As the object information, a LAN 
transmission channel ("LAN Port Number"), a flag 
showing whether scramble is used ("Scramble Flag"), a 
field for defining an offset value for changing the 
processing priority when a terminal is overloaded 
(XGD Offset), and an identifier (Media Type) for identi- 
fying a type of the media (picture or audio) to be trans- 
mitted are described. 

[01 72] In the case of the example in Figure 16(b). AL 
(in this case, defined as additional information neces- 
sary to decode pictures for one frame) is added to con- 
trol decoding of ES (in this case, defined as a data string 
corresponding to pictures for one frame). As AL infor- 
mation, the following are defined. 

(1) Random Access Flag (flag showing whether to 
be independently reproducible, true for an intra- 
frame encoded picture frame) 

(2) Presentation Time stamp (time displayed by 
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frame) 

(3) CGD Priority (Value of priority for deciding 
processing priority when terminal is overloaded) 

[0173] The example shows a case of transmitting the 5 
data string for one frame by using RTF (protocol for 
transmitting continuous media through internet, Real- 
time Transfer Protocol). "AL Reconfiguration" is a trans- 
mission expression for changing the maximum value 
that can be expressed by the above AL io 
[0174] The example in Figure 1 6(b) makes it possible 
to express up to 2 bits as "Random Access Flag Max 
Bit". For example, when there is no bit, Random Access 
Flag is not used. When there are two bits, the maximum 
value is equal to 3. is 
[0175] Moreover, the expression with a real number 
part and a mantissa part is allowed (e.g. 3 *6). When no 
data is set an operation under the state decided by 
default is allowed. 

[0176] In Figure 17, "Setup Request" shows a trans- 20 
mission expression for transmitting a setup time. "Setup 
Request" is transmitted before a program is transmitted, 
a transmission channel number ("Logical Channel 
Number") to be transmitted, a program ID ("execute 
Program Number") to be executed, a data ID ("data 2s 
Number") to be used, and the ID of a command ("exe- 
cute Command Number") to be executed are made to 
correspond to each other and transmitted to a receiving 
terminal. Moreover, an execution authorizing flag 
("flag"), a counter ("counter") describing whether to start 30 
execution when receiving Setup Request how many 
times, and a timer value ("timer") showing whether to 
start execution after how many hours pass can be used 
as other expression methods by making them corre- 
spond to transmission channel numbers. 35 
[0177] Rewriting of AL information and securing of 
rise time of Group Mux are listed as examples of 
requests to be demanded. 

[0178] Figure 18 is an illustration for explaining a 
transmission expression for communicating whether to 40 
use the AL described for Figure 16(b) from a transmit- 
ting terminal to a receiving terminal ("Control AL defini- 
tion"). 

[0179] In Figure 18, if "Random Access Flag Use" is 
true, Random Access Rag is used. If not, it is not used, as 
It is possible to transmit the AL change notification as 
control information through a transmission channel sep- 
arate from that of data or transmit it through the trans- 
mission channel same as that of data together with the 
data. so 
[0180] A decoder program is listed as a program to be 
executed. Moreover, a setup request can be used for 
broadcasting and communication. Furthermore, which 
item serving as control information is used as Al infor- 
mation is designated to a receiving terminal in accord- 55 
ance with the above request. Furthermore, it is possible 
to designate which item is used as communication 
header, which item is used as AL information and which 



item is used as control information to a receiving termi- 
nal. 

[01 81 ] Figure 1 9(a) shows the example of a transmis- 
sion expression for changing the structure of header 
information (data control information, transmission con- 
trol information, and control information) to be transmit- 
ted by using an information frame identifier ("header 
ID") between transmitting and receiving terminals in 
accordance with the purpose. 
[0182] In Figure 19(a), "class ES header" separates 
the structure of the data control information to be trans- 
mitted through a transmission channel same as that of 
data from that of the information with which transmis- 
sion control information is transmitted between trans- 
mitting and receiving terminals in accordance with an 
information frame identifier. 

[0183] For example, only the item of "buffer Size ES" 
is used when the value of "header ID" is 0 but the item 
of "reserved" is added when the value of "header ID" is 
1. 

[0184] Moreover, by using a default identifier ("use 
Header Extension"), it is decided whether to use a 
default-type information frama When "use Header 
Extension" is true, an item in an if -statement is used. It 
is assumed that these pieces of structural information 
are previously decided between transmitting and receiv- 
ing terminals. Furthermore, it is possible to use a struc- 
ture for using either of an information frame identifier 
and a default identifier. 

[0185] in Figure 19(b), "AL configuration" shows an 
example for changing the structure of control informa- 
tion to be transmitted through a transmission channel 
different from that of data between transmitting and 
receiving terminals in accordance with the purpose. The 
usage of an information frame identifier and that of a 
default identifier are the same as the case of Figure 
19(a). 

[0186] In the case of the present invention, methods 
for realizing a system for simultaneously synthesizing 
and displaying a plurality of pictures and a plurality of 
audio are specifically described from the following view- 
points. 

(1) A method for transmitting (communicating and 
broadcasting) a picture and an audio through a plu- 
rality of logical transmission lines and controlling 
them: Particularly, a method for respectively trans- 
mitting control information and data through an 
independent logical transmission line is described. 

(2) A method for dynamically changing header 
information (AL information) added to the data for a 
picture or audio to be transmitted. 

(3) A method for dynamically changing communica- 
tion header information added for transmission. 

Specifically, for Items (2) and (3), a method for 
uniting and controlling the information overlapped 
in AL information and communication header and a 
method for transmitting AL information as control 
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information are described. 

(4) A method for dynamically multiplexing and sep- 
arating a plurality of logical transmission fines and 
transmitting information. 

A method for economizing the number of chan- s 
nels of transmission lines and a method for realiz- 
ing efficient multiplexing are described. 

(5) A method for reading a program or data and 
transmitting pictures and audio considering a rise 
time. Moreover, a method for reducing an apparent 10 
setup time for various functions and purposes is 
described. 

(6) A method for transmitting a picture or audio for 
zapping. 

[0187] The present invention is not restricted to only 
synthesis of two-dimensional pictures. It is also possible 
to use an expression method of combining a two-dimen- 
sional picture with a three-dimensional picture or 
include a picture synthesizing method for synthesizing a 20 
plurality of pictures so that they are adjacent to each 
other like a wide-visual-field picture (panoramic picture). 
[0188] Moreover, the present invention does not pur- 
pose only such communication systems as bidirectional 
CATV and B-ISDN. For example, it is possible to use 25 
radio waves (e.g. VHF band or UHF band) or a broad- 
casting satellite for transmission of pictures and audio 
from a center-side terminal to a home-side terminal and 
an analog telephone line or N-ISDN for transmission of 
information from a home-side terminal to a center-side 30 
terminal (it is not always necessary that pictures, audio, 
and data are multiplexed) . 

[01 89] Moreover, it is possible to use a communication 
system using radio such as IrDA, PHS (Personal Handy 
Phone), or radio LAN. Furthermore, a purposed termi- 35 
nal can be a portable terminal such as a portable infor- 
mation terminal or a desktop terminal such as a setup 
BOX or personal computer. Furthermore, a video tele- 
phone, multipoint monitoring system, multimedia data- 
base retrieval system, and game are listed as 40 
application fields. The present invention includes not 
only a receiving terminal but also a server and a 
repeater to be connected to a receiving terminal. 
[0190] Furthermore, in the case of the above exam- 
ples, a method for avoiding the overlap of the (cornmu- 45 
nication) header of RTP with AL information and a 
method for extending the communication header of RTP 
or AL information are described. However, it is not 
always necessary for the present invention to use RTP 
For example, it is also possible to newly define an origi- so 
nal communication header or AL information by using 
UDP or TCP. Though an internet profile uses RTP 
sometimes, a multifunctional header such as RTP is not 
defined for a Raw profile. There are four types of con- 
cepts about AL information and communication header 55 
as described above. 

[0191] Thus, by dynamically deciding the information 
frame of data control information, transmission control 



information, or control information used by the transmit- 
ting and receiving terminals (e.g. information frame 
including the sequence of information to be added and 
the number of bits for firstly assigning a random access 
flag as 1 -bit flag information and secondly assigning 16 
bits in the form of a sequence number), it is possible to 
change only an information frame corresponding to the 
situation in accordance with the purpose or transmis- 
sion line. 

[01 92] The frame of each piece of information can be 
any one of the frames already shown in Figures 6(a) to 
6(d) and in the case of RTP, the data control information 
(AL) can be the header information for each medium 
(e.g. in the case of H.263, the header information of the 
video or that of the payload intrinsic to H.263), transmis- 
sion control information can be the header information 
of RTP, and control information can be the information 
for controlling RTP such as RTCP. 
[01 93] Moreover, in the case of a publicly-known infor- 
mation frame previously set between transmitting and 
receiving terminals, by providing a default identifier for 
showing whether to process information by transmitting 
and receiving for data control information, transmission 
control information, and control information (information 
transmitted through a packet different from that of data 
to control terminal processing) respectively, it is possi- 
ble to know whether information frames are changed. 
By setting the default identifier and communicating the 
changed content (such as change of time stamp infor- 
mation from 32 to 16 bits) only when change is per- 
formed in accordance with the method shown in Figure 
1 6, it is prevented to transmit unnecessary configuration 
information even when frame information of information 
is not changed. 

[0194] For example, the following two methods are 
considered to change information frames of data control 
information. First, to describe a method for changing 
information frames of data control information in data, 
the default identifier (to be" written in a fixed region or 
position) of the information present in the data 
described for the information frame of data control infor- 
mation is set and then, information frame change con- 
tents are described. 

[0195] To change information frames of data control 
information by describing a method for changing only 
the information frames of data in the control information 
(information frame control information) as another 
method, a default identifier provided for control informa- 
tion is set, the contents of the information frames of the 
data control information to be changed are described, 
and it is communicated to a receiving terminal in 
accordance with ACK/Reject and confirmed that the 
information frames of the data control information are 
changed and thereafter, the data in which information 
frames are changed is transmitted. Information frames 
of transmission control information and control informa- 
tion can be also changed in accordance with the above 
two methods (Figure 19). 
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[0196] More specifically, though the header informa- 
tion of MPEG2 is fixed, by providing a default identifier 
for a program map table (defined by PSJ) for relating the 
video stream of MPEG2-Ts (transport stream) with the 
audio stream of it and defining a configuration stream in 5 
which a method for changing frames of the information 
for the video stream and audio stream is described, it is 
possible to first interpret the configuration stream and 
then, interpret the headers of the video and audio 
streams in accordance with the content of the configura- 
tion stream when the default identifier is set It is possi- 
ble for the configuration stream to have the contents 
shown in Figure 19. 

[0197] The contents (transmltted-format information) 
of the present invention about a transmission method 
and/or a structure of the data to be transmitted corre- 
spond to, for example, an information frame in the case 
of the above embodiment. 

[0198] Moreover, for the above embodiments, a case 
of transmitting the contents to be changed concerned 
with a transmission method and/or the structure of the 
data to be transmitted is mainly described. However, it is 
also possible to use a structure for transmitting only the 
identifier for the contents. In this case, as shown in Fig- 
ure 44, it is also possible to use an audio-video transmit- 
ter provided with (1) transmitting means 5001 for 
transmitting the content concerned with a transmission 
method and/or the structure of the data to be transmit- 
ted or an identifier showing the content as the transmit- 
ted-format information through the transmission line 
same as that of the data to be transmitted or a transmis- 
sion line different from the former transmission line and 
(2) storing means 5002 for storing a plurality of types of 
the contents concerned with the transmission method 
and/or the structure of the data to be transmitted and a 
plurality of types of identifiers for the contents, in which 
the identifiers are included in at least one of the data 
control information, transmission control information, 
and information for controlling terminal-side processing. 
Moreover, as shown in Figure 45. it is possible to use an 
audio-video receiver provided with receiving means 
5101 for receiving the transmission format information 
transmitted from the audio-video transmitter and trans- 
mission information interpreting means 5102 for inter- 
preting the received transmission format information. 
Furthermore, the audio-video receiver can be consti- 
tuted with storing means 5103 for storing a plurality of 
types of contents concerned with the transmission 
method and/or the structure of the data to be transmit- 
ted and a plurality of types of identifiers for the contents 
to use the contents stored in the storing means to inter- 
pret the contents of the identifiers when receiving the 
identifiers as the transmission format information. 
[0199] More specifically, by preparing a plurality of 
types of information frames previously determined 
between transmitting and receiving terminals and trans- 
mitting identifiers for the above information frames and 
information frame identifiers for a plurality of types of 



data control information, a plurality of types of transmis- 
sion control information, and a plurality of types of con- 
trol information (information-frame control information) 
together with data or as control information, H is possi- 
ble to identify a plurality of types of data control informa- 
tion, a plurality of types of transmission control 
information, and a plurality of types of control informa- 
tion and optionally select the information frame of each 
type of information in accordance with the type of a 
medium to be transmitted or the size of a transmission 
line. Identifiers of the present invention correspond to 
the above information frame identifiers. 
[0200] It is possible to read and interpret these infor- 
mation identifiers and default identifiers even if informa- 
tion frames are changed at a receiving-side terminal by 
adding the identifiers to a predetermined fixed-length 
region or predetermined position of the information to 
be transmitted. 

[0201] Moreover, in addition to the structures 
described for the above embodiments, it is possible to 
use a structure for temporarily selecting the caption pic- 
ture of a program to be looked and listened by the user 
and showing it for the user when it takes a lot of time to 
set up a necessary program or data by using a broad- 
casting channel for broadcasting only the heading pic- 
tures of pictures broadcasted through a plurality of 
channels and switching programs to be looked and lis- 
tened by the user. 

[0202] As described above, the present invention 
makes H possible to change frames of the information 
corresponding to the situation in accordance with the 
purpose or transmission line by dynamically determin- 
ing the frame of data control information, transmission 
control information, or control information used by trans- 
mitting and receiving terminals. 
[0203] Moreover, it is possible to know whether infor- 
mation frames are changed by providing a default iden- 
tifier for showing whether to transit or receive and 
process information by a publicly-known information 
frame previously set between transmitting and receiving 
terminals for data control information, transmission con- 
trol information, and control information respectively 
and it is possible to prevent unnecessary configuration 
information from being transmitted even if information 
frames of information are not changed by setting a 
default identifier and communicating changed contents 
only when change is performed. 
[0204] Furthermore, it is possible to identify a plurality 
of types of data control information, a plurality of types 
of transmission control information, and a plurality of 
types of control information by preparing a plurality of 
information frames previously determined between 
transmitting and receiving terminals and transmitting 
information frame identifiers for identifying a plurality of 
types of data control information, a plurality of types of 
transmission control information, and a plurality of types 
of control information together with data or as control 
information and optionally select the information frame 
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of each type of information in accordance with the type 
of a medium to be transmitted or the size of a transmis- 
sion line. 

[0205] It is possible to read and interpret these infor- 
mation identifiers and default identifiers even if informa- s 
tion frames are changed at a receiving-side terminal by 
adding the identifiers to a predetermined fixed-length 
region or predetermined position of the information to 
be transmitted. 

[0206] Embodiments of the present invention are to 
described below by referring to the accompanying draw- 
ings. 

[0207] In this case, any one of the above-described 
problems (B1) to (B3) is solved. 

[0208] A "picture ( or video)" used for the present 15 
invention includes both a static picture and a moving 
picture. Moreover, a purposed picture can be a two- 
dimensional picture such as a computer graphics (CG) 
or three-dimensional picture data constituted with a 
wire-frame model. 20 
[0209] Figure 25 is a schematic block diagram of the 
picture encoder and a picture decoder of an embodi- 
ment of the present invention. 

[021 0] A transmission control section 40 11 for trans- 
mitting or recording various pieces of encoded informa- 2s 
tion is means for transmitting the information for coaxial 
cable, CATV, LAN, or modem. A picture encoder 4101 
has a picture encoding section 4012 for encoding pic- 
ture information such as H.263, MPEG1/2, JPEG, or 
Huffman encoding and the transmission control section 30 
401 1. Moreover, a picture decoder 4102 has an output 
section 40 1 6 constituted with a reception control section 
4013 for receiving various pieces of encoded informa- 
tion, a picture decoding section 4014 for decoding vari- 
ous pieces of received picture information, a picture 35 
synthesizing section 4015 for synthesizing one decoded 
picture or more, and an output section 4016 constituted 
with a display and a printer for outputting pictures. 
[021 1 ] Figure 26 is a schematic block diagram of the 
audio encoder and an audio decoder of an embodiment 40 
of the present invention. 

[0212] An audio encoder(sound encorder) 4201 is 
constituted with a transmission control section 4021 for 
transmitting or recording various pieces of encoded 
information and an audio encoding section 4022 for 45 
encoding such audio information such as G.721 or 
MPEG1 audio. Moreover, an audio decoder(a sound 
decoder) 4202 is constituted with a reception control 
section 4023 for receiving various pieces of encoded 
information, an audio decoding section 4024 for decod- so 
ing the above pieces of audio information, an audio syn- 
thesizing section (a sound synthesizing section)4025 
for synthesizing one decoded audio or more, and output 
means 4026 for outputting audio. 

[021 3] Time-series data for audio or picture is specif i- 55 
cally encoded or decoded by the above encoder or 
decoder. 

[0214] The communication environments in Figures 



25 and 26 can be a communication environment in 
which a plurality of logical transmission lines can be 
used without considering multiplexing means like the 
case of internet or a communication environment in 
which multiplexing means must be considered like the 
case of an analog telephone or satellite broadcasting. 
Moreover, a system for bilaterally transferring a picture 
or audio between terminals like a video telephone or 
video conference or a system for broadcasting a broad- 
casting-type picture or audio on satellite broadcasting, 
CATV, or internet is listed as a terminal connection sys- 
tem. 

[021 5] Moreover, a method for synthesizing a picture 
and audio can be defined by describing a picture and an 
audio, structural information for a picture and an audio 
(display position and display time), an audio-video 
grouping method, a picture display layer (depth), and an 
object ID (ID for identifying each object such as a pic- 
ture or audio) and the relation between the attributes of 
them with a script language such as JAVA, VRML, or 
MHEG. A script describing a synthesizing method is 
obtained from a network or local memory. 
[021 6] Moreover, it is possible to constitute a transmit- 
ting or receiving terminal by optionally combining an 
optional number of picture encoders, picture decoders, 
audio encoders, and audio decoders. 
[0217] Figure 27(a) is an illustration for explaining a 
priority adding section and a priority deciding section for 
controlling the priority for processing under overload. A 
priority adding section 31 for deciding the priority for 
processing encoded information under overload in 
accordance with a predetermined criteria by an encod- 
ing method such as H.263 or G.723 and relating the 
encoded information to the decided priority is provided 
for the picture encoder 4101 and audio encoder 4201. 
[0218] The criteria for adding a priority are scene 
change in the case of a picture and audio and audioless 
blocks in the case of a picture frame, stream, or audio 
designated by an editor oruser. 
[021 9] A method for adding a priority to a communica- 
tion header and a method for embedding a priority in the 
header of a bit stream to be encoded of a video or audio 
under encoding are considered as priority adding meth- 
ods for defining a priority under overload. The former 
method makes it possible to obtain the information con- 
cerned with & priority without decoding the information 
and the latter method makes it possible to independ- 
ently handle a single bit stream without depending on a 
system. 

[0220] As shown in Figure 27(b), when priority infor- 
mation is added to a communication header and one 
picture frame (e.g. intra-frame encoded l-f rame or inter- 
frame encoded P- or B-frame) is divided into a plurality 
of transmission packets, a priority is added only to a 
communication header for transmitting the head, of a 
picture frame accessible as single information in the 
case of a picture (when priorities are equal in the same 
picture frame, it is possible to assume that the priorities 
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are not changed until the head of the next accessible 
picture frame appears). 

[0221] Moreover, in the case of a decoder, a priority 
deciding section 32 for deciding a processing method is 
provided for the picture decoder 4102 and audio 
decoder 4202 in accordance with the priorities of vari- 
ous pieces of encoded information received under over- 
load. 

[0222] Figures 28(a) to 28(c) are illustrations for 
explaining the grading for adding a priority. Decoding is 
performed by using two types of priorities for deciding 
the priority for processing under overload at a terminal. 
[0223] That is, a stream priority (Stream Priority; inter- 
time-series-data priority) for defining the priority for 
processing under overload in bit streams such as pic- 
ture and audio and a frame priority (Frame Priority; 
intra-time-series-data priority) for defining the priority for 
processing under overload in frames such as picture 
frames in the same stream are defined (see Figure 
28(a)). 

[0224] The former stream priority makes it possible to 
handle a plurality of videos or audios. The latter frame 
priority makes it possible to add a different priority to a 
picture scene change or the same intra-f rame encoded 
picture frame (l-frame) in accordance with the intention 
of an editor. 

[0225] A value expressed by the stream priority repre- 
sents a case of handling it as a relative value and a case 
of handling it as an absolute value (see Figures 28(b) 
and 28(c)). 

[0226] The stream and frame priorities are handled by 
a repeating terminal such as a router or gateway on a 
network and by transmitting and receiving terminate in 
the case of a terminal. 

[0227] Two types of methods for expressing an abso- 
lute value or relative value are considered. One of them 
is the method shown in Figure 28(b) and the other of 
them is the method shown in Figure 28(c). 
[0228] In Figure 28(b), the priority of an absolute value 
is a value showing the sequence in which picture 
streams (video streams) or audio streams added by an 
editor or mechanically added are processed (or to be 
processed) under overload (but not a value considering 
the load fluctuation of an actual network or terminal). 
The priority of a relative value is a value for changing the 
value of an absolute priority in accordance with the load 
of a terminal or network. 

[0229] By dividing a priority into a relative value and 
an absolute value to control the values and thereby 
changing only relative values at the transmitting side or 
by a repeater in accordance with the load fluctuation of 
a network or the like, it is possible to record the value of 
an absolute value into a hard disk or VTR while leaving 
the absolute priority added to a video or audio stream. 
Thus, when the value of the absolute priority is 
recorded, it is possible to reproduce a picture or audio 
that is not influenced by the load fluctuation of a network 
or the like. Moreover, it is possible to transmit a relative 



or absolute priority through a control channel independ- 
ently of data. 

[0230] Moreover, in Figure 28(b), it is possible to fine 
the grading compared to a stream priority and handle a 

5 frame priority for defining the priority for frame process- 
ing under overload as the value of a relative priority or 
handle it as the value of an absolute priority. For exam- 
ple, by describing an absolute frame priority in encoded 
picture information and descrfoing a relative frame prior- 

10 ity corresponding to the absolute priority added to the 
picture frame in the communication header of a commu- 
nication packet for transmitting encoded information in 
order to reflect the load fluctuation of a network or termi- 
nal, it is possible to add a priority corresponding to the 

is load of a network or terminal even at a frame level while 
leaving an original priority. 

[0231] Moreover, it is possible to transmit a relative 
priority by descrfoing the relation with a frame not in a 
communication header but in a control channel inde- 

20 pendentiy of data. Thereby, it is possible to record data 
into a hard disk or VTR while leaving an absolute priority 
originally added to a picture or audio stream. 
[0232] Furthermore, in Figure 28(b). when reproduc- 
ing data at a receiving terminal while transmitting the 

25 data through a network without recording the data at the 
receiving terminal, it is possible to compute the value of 
an absolute priority and that of a relative priority at 
frame and stream levels at the transmitting side and 
thereafter transmit only absolute values because it is 

30 unnecessary to control absolute and relative values by 
separating them from each other at a receiving terminal. 
[0233] in Figure 28(c), the priority of an absolute value 
is a value uniquely determined between frames 
obtained from the relation between Stream Priority and 

3$ Frame Priority. The priority of a relative value is a value 
showing the sequence in which picture streams or audio 
streams added by an editor or mechanically added are 
processed (or to be processed) under overload. In the 
case of the example in Figure 28(c), the frame priority of 

40 a picture or audio stream (relative: relative value) and 
the stream priority for each stream are added. 
[0234] An absolute frame priority (absolute; absolute 
value) is obtained from the sum of a relative frame prior- 
ity and a stream priority (That is, absolute frame priority 

45 = relative frame priority + stream priority). To obtain an 
absolute frame priority, it is also possible to use a sub- 
tracting method or a constant-multiplying method. 
[0235] An absolute frame priority mainly uses a net- 
work. This is because the expression using an absolute 

so value does not require the necessity for deciding a pri- 
ority for each frame through a repeater such as a router 
or gateway by considering Stream Priority and Frame 
Priority. By using the absolute frame priority, such 
processing as disuse of a frame by a repeater is simpli- 

55 fied. 

[0236] Moreover, it can be expected to apply a relative 
frame priority mainly to an accumulation system for per- 
forming recording or editing. In the case of an editing 
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operation, a plurality oJ picture and audio streams may 
be handled at the same time. In this case, the number of 
picture streams or the number of frames that can be 
reproduced may be limited depending on the load of a 
terminal or network. s 
[0237] In the above case, it is unnecessary to recalcu- 
late every Frame Priority differently from the case in 
which an absolute value is expressed only by separat- 
ing Stream Priority from Frame Priority, that is, only by 
changing Stream Priority of a stream which an editor io 
wants to preferentially display or a user wants to see. 
Thus, it is necessary to use an absolute expression or a 
relative expression in accordance with the purpose. 
[0238] By describing whether to use a stream priority 
as a relative value or absolute value, it is possible to is 
effectively express a priority for transmission and accu- 
mulation. 

[0239] In the case of the example in Figure 28(b), it is 
differentiated by following a stream priority that the 
value expressed by the stream priority is a relative value 20 
or absolute value by using a flag or identifier for 
expressing whether the value expressed by the stream 
priority is an absolute value or relative value. In the case 
of a frame priority, a flag or identifier is unnecessary 
because a relative value is described in a communica- 25 
tion header and an absolute value is described in an 
encoded frame. 

[0240] In the case of the example in Figure 28(c). a 
flag or identifier for identifying whether a frame priority is 
an absolute value or relative value is used. In the case 30 
of an absolute value, the frame priority is a priority cal- 
culated in accordance with a stream priority and a rela- 
tive frame priority and therefore, the calculation is not 
performed by a repeater or terminal. Moreover, when 
the calculation formula is already known at a terminal, it 35 
is possible to inversely calculate a relative frame priority 
from an absolute frame priority and a stream priority. 
For example, it is also possible to obtain the absolute 
priority (Access Unit Priority) of a packet to be transmit- 
ted from the relational expression 40 
[0241 ] "Access Unit Priority = stream priority - frame 
priority". 

In this case, it is also possible to express the frame pri- 
ority as a degradation priority because it is obtained 
after being subtracted from the stream priority. 45 
[0242] Moreover, it is also possible to control data 
processing by relating one stream priority or more to the 
priority for processing of the data passing through the 
logical channel of TCP/IP (port No. of LAN). 
[0243] Furthermore, it is expected that the necessity so 
for retransmission can be reduced by assigning a 
stream priority or frame priority lower than that of a 
character or control information to a picture or audio. 
This is because no problem occurs in most cases even 
if a part of a picture or audio is lost 55 
[0244] Figure 29 is an illustration for explaining a 
method for assigning a priority to multi-resolution video 
data. 



[0245] When one stream is constituted with a plurality 
of substreams , it is possible to define a substream 
processing method by adding a stream priority to the 
substreams and describing a logical sum or logical 
product under accumulation or transmission. 
[0246] In the case of a wavelet, it is possible to decom- 
pose one picture frame into a plurality of different-reso- 
lution picture frames. Moreover, even in the case of a 
DCT-base encoding method, it is possible to decom- 
pose one picture frame into a plurality of different-reso- 
lution picture frames by dividing the picture frame into a 
high-frequency component and a low-frequency com- 
ponent and encoding them. 

[0247] In addition to stream priorities added to a plu- 
rality of picture streams constituted with a series of 
decomposed picture frames, the relation between pic- 
ture streams is defined with AND (logical product) and 
OR (logical sum) in order to describe the relation. Spe- 
cifically, when the stream priority of a stream A is 5 and 
that of a stream B is 10 (the smaller a numerical value 
gets, the higher a priority becomes), the relation 
between picture streams is defined that the stream B is 
disused in the case of disuse of stream data depending 
on the priority but the stream B is transmitted and proc- 
essed without being disused even if the priority of the 
stream B is lower than the priority of a threshold in the 
case of AND by describing the relation between 
streams. 

[0248] Thereby, relevant streams can be processed 
without being disused. In the case of OR, it is defined 
that relevant streams can be disused. It is possible to 
perform disuse processing at a transmitting or receiving 
terminal or a repeating terminal as ever. 
[0249] Moreover, when the same video clip is 
encoded to 24 Kbps and 48 Kbps respectively as an 
operator for relational description, there is a case in 
which either 24 or 48 Kbps may be reproduced (exclu- 
sive logical sum EX-OR as relational description). 
[0250] When the priority of the former is set to 10 and 
that of the latter is set to 5, a user can reproduce the lat- 
ter in accordance with a priority or select the latter with- 
out following the priority. 

[0251 ] Figure 30 is an illustration for explaining a com- 
munication payload constituting method. 
[0252] When constituted with a plurality of sub- 
streams, disuse at a transmission packet level becomes 
easy by, for example, constituting transmission packets 
starting with, for example, one having the highest prior- 
ity in accordance with a stream priority added to a sub- 
stream. Moreover, disuse at a communication packet 
level becomes easy by fining grading and uniting the 
information for objects respectively having a high frame 
priority and thereby constituting a communication 
packet. 

[0253] By relating the sliced structure of a picture to a 
communication packet, return of a missing packet 
becomes easy. That is, by relating the sliced structure of 
a video to a packet structure, a re-sync marker for 
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^synchronization is unnecessary. Unless a sliced 
structure coincides with the structure of a communica- 
tion packet, it is necessary to add a. re-sync marker 
(marker for making a returning position known) so that 
^synchronization can be performed if information is 
damaged due to a missing packet). 
[0254] In accordance with the above-mentioned, it is 
considered to apply a high error protection to a commu- 
nication packet having a high priority. Moreover, the 
sliced structure of a picture represents the unit of col- 
lected picture information such as GOB or MB. 
[0255] Figure 31 is an illustration for explaining a 
method for relating data to communication payload. By 
transmitting a method for relating a stream or object to a 
communication packet together with control information 
or data, it is possible to generate an optional data format 
in accordance with the communication state or purpose. 
For example, in the case of RTP (Real time Transfer 
Protocol), the payload of RTP is defined for each encod- 
ing to be handled. The format of the existing RTP is 
fixed. In the case of H.263, as shown in Figure 31 , three 
data formats from Mode A to Mode C are defined. In the 
case of H.263, a communication payload purposing a 
multi-resolution picture format is not defined. 
[0256] In the case of the example in Figure 31 , Layer 
No. and the above relational description (AND, OR) are 
added to the data format of Mode A and defined. 
[0257] Figure 32 is an illustration for explaining the 
relation between frame priority, stream priority, and 
communication packet priority. 
[0258] Moreover, Figure 32 shows an example of 
using a priority added to a communication packet on a 
transmission line as a communication packet priority 
and relating a stream priority and a frame priority to the 
communication packet priority. 
[0259] Generally, in the case of communication using 
IP, it is necessary to transmit data by relating a frame 
priority or stream priority added to picture or audio data 
to the priority of a low-order IP packet. Because the pic- 
ture or audio data is divided into IP packets and trans- 
mitted, it is necessary to relate priorities to each other. 
In the case of the example in Figure 32, because the 
stream priority takes values from 0 to 3 and the frame 
priority takes values from 0 to 5, high-order data can 
take priorities from 0 to 1 5. 

[0260] In the case of IPv6, priorities (4 bits) from 0 to 
7 are reserved for congestion-controlled traffic. Priori- 
ties from 8 to 1 5 are reserved for real-time communica- 
tion traffic or not-congestion-controlled traffic. Priority 
15 is the highest priority and priority 8 is the lowest pri- 
ority. This represents the priority at the packet level of IP 
[0261] in the case of data transmission using IP, it is 
necessary to relate high-order priorities from 0 to 15 to 
low-order IP priorities from 8 to 1 5. To relate priorities to 
each other, it is possible to use a method of clipping 
some of high-order priorities or relate priorities to each 
other by using a performance function. Relating of high- 
order data with a low-order IP priority is performed at a 
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repeating node (router or gateway) or transmitting and 
receiving terminals. 

[0262] Transmitting means is not restricted to only IP. 
It is possible to use a transmission packet having a flag 
5 showing whether it can be disused like TS (transport 
stream) of ATM or MPEG2. 

[0263] The frame priority and stream priority having 
been described so far can be applied to a transmitting 
medium or data-recording medium. It is possible to use 
10 a floppy disk or optical disk as a data-recording 
medium. 

[0264] Moreover, it is possible to use not only the 
floppy disk or optical disk but also a medium such as an 
IC card or ROM cassette as long as a program can be 
15 recorded in the medium. Furthermore, it is possible to 
use an audio-video repeater such as a router or gate- 
way for relaying data. 

[0265] Furthermore, preferential retransmission is 
realized by deciding time-series data to be retransmit- 

20 ted in accordance with the information of Stream Prior- 
ity (inter-time-series-data priority) or Frame Priority 
(intra-time-series-data priority). For example, when 
decoding is performed at a receiving terminal in accord- 
ance with priority information, it is possible to prevent a 

25 stream or frame that is not an object for processing from 
being retransmitted. 

[0266] Furthermore, separately from a present priority 
to be processed, it is possible to decide a stream or 
frame having a priority to be retransmitted in accord- 

30 ance with the relation between retransmission fre- 
quency and successful transmission frequency. 
[0267] Furthermore, in the case of a transmitting-side 
terminal, preferential transmission is realized by decid- 
ing time-series data to be transmitted in accordance 

35 with the information of Stream Priority (inter-time- 
series-data priority) or Frame Priority (intra-time-series- 
data priority). For example, by deciding the priority of a 
stream or frame to be transmitted in accordance with an 
average transfer rate or retransmission frequency, it is 

40 possible to transmit an adaptive picture or audio even 
when a network is overloaded. 
[0268] The above embodiment is not restricted to two- 
dimensional -picture synthesis. It is also possible to use 
an expression method obtained by combining a two- 

4$ dimensional picture with a three-dimensional picture or 
include a picture-synthesizing method for synthesizing 
a plurality of pictures so as to be adjacent to each other 
like a wide-visual-field picture (panorama picture). 
Moreover, communication systems purposed by the 

so present invention are not restricted to bidirectional 
CATV or B-ISDN. For example, transmission of pictures 
and audio from a center-side terminal to a house-side 
terminal can use radio waves (e.g. VHF band or UHF 
band) or satellite broadcasting and information origina- 

55 tion from the house-side terminal to the center-side ter- 
minal can use an analog telephone line or N-ISDN (it is 
not always necessary that pictures, audio, or data are 
multiplexed). Moreover, it is possible to use a communi- 
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cation system using radio such as an IrDA, PHS (Per- 
sonal Handy Phone) or radio LAN. 
[0269] Furthermore, a purpose terminal can be a port 
able terminal such as a portable information terminal or 
a desktop terminal such as a set-top BOX or personal s 
computer. 

[0270] As described above, the present invention 
makes it easy to handle a plurality of video streams and 
a plurality of audio streams and mainly synchronise and 
reproduce important scene cut together with audio by 10 
reflecting the intention of an editor.. 
[0271] An embodiment of the present invention is 
described below by referring to the accompanying draw- 
ings. 

[0272] The embodiment described below solves any 15 
one of the above problems (C1) to (C3). 
[0273] Figure 33 shows the structure of the transmitter 
of the first embodiment. Symbol 2101 denotes a picture- 
input terminal and the size of a sheet of picture has 144 
pixels by 176 pixels. Symbol 2102 denotes a video 20 
encoder that is constituted with four components 1021 , 
1022, 1023. and 1024 (see Recommendation H.261). 
[0274] Symbol 1 02 1 denotes a switching unit for divid- 
ing an input picture into macroblocks (a square region of 
16 pixels by 16 pixels) and deciding whether to intra- 2s 
encode or inter-encode the blocks and 1022 denotes 
movement compensating means for generating a move- 
ment compensating picture in accordance with the local 
decoded picture which can be calculated in accordance 
with the last-time encoding result, calculating the differ- 30 
ence between the movement compensating picture and 
an input picture, and outputting the result in macrob- 
locks. Movement compensation includes halfpixel pre- 
diction having a long processing time and fullpixel 
prediction having a short processing time. Symbol 1 023 35 
denotes orthogonal transforming means for applying 
DOT transformation to each macroblock and 1024 
denotes variable-length-encoding means for applying 
entropy encoding to the DCT transformation result and 
other encoded information. 40 
[0275] Symbol 2103 denotes counting means for 
counting execution frequencies of four components of 
the video encoder 2102 and outputting the counting 
result to transforming means every input picture. In this 
case, the execution frequency of the halfpixel prediction as 
and that of the fullpixel prediction are counted from the 
movement compensating means 1 022. 
[0276] Symbol 21 04 denotes transforming means for 
outputting the data string shown in Figure 34. Symbol 
2105 denotes transmitting means for multiplexing a var- so 
iable-length code sent from the video encoder 2102 and 
a data string sent from the transforming means 2104 
into a data string and outputting the data string to a data 
output terminal 210SL 

[0277] According to the above structure, it is possible ss 
to transmit the execution frequencies of indispensable 
processing (switching unit 1021. orthogonal transform- 
ing means 1023. and variable-length encoding means 
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1024) and dispensable processing (movement compen- 
sating means 1022) to a receiver. 
[0278] The transmitter of the first embodiment corre- 
sponds to claim 68. 

[0279] Figure 40 is a flowchart of the transmitting 
method of the second embodiment 
[0280] Because operations of this embodiment are 
similar to those of the first embodiment, corresponding 
elements are added. A picture is input in step 801 (pic- 
ture input terminal 2101) and the picture is divided into 
macroblocks in step 802. Hereafter, processings from 
step 803 to step 806 are repeated until the processing 
corresponding to every macroblock is completed in 
accordance with the conditional branch in step 807. 
Moreover, when each processing is executed so that 
frequencies of the processings from step 803 to step 
806 can be recorded in specific variables, a correspond- 
ing variable is incremented by 1 . 
[0281] First it is decided whether to intra-encode or 
inter-encode a macroblock to be processed in step 803 
(switching unit 1021). When inter-encoding the macrob- 
lock, movement compensation is performed in step 804 
(movement compensating means 1022). Thereafter. 
DCT transformation and variable-length encoding are 
performed in steps 805 and 806 (orthogonal transform- 
ing means 1023 and variable-length encoding means 
1024 ). When processing for every macroblock is com- 
pleted (in the case of Yes in step 807), the variable 
showing the execution frequency corresponding to each 
processing is read in step 808, the data string shown in 
Figure 2 is generated, and the data string and a code 
are multiplexed and output. The processings from step 
801 to step 808 are repeatedly executed as long as 
input pictures are continued. 

[0282] The above structure makes it possible to trans- 
mit the execution frequency of each processing. 
[0283] The transmitting method of the second embod- 
iment corresponds to claim 67. 
[0284] Rgure 35 shows the structure of the receiver of 
the third embodiment. 

[0285] In Figure 35, symbol 307 denotes an input ter- 
minal for inputting the output of the transmitter of the 
first embodiment and 301 denotes receiving means for 
fetching a variable-length code and a data string 
through inverse multiplexing in accordance with the out- 
put of the transmitter of the first embodiment and out- 
putting them. In this case, it is assumed that the time 
required to receive the data for one sheet is measured 
and also output. 

[0286] Symbol 303 denotes a decoder for a video 
using a variable-length code as an input, which is con- 
stituted with five components. Symbol 3031 denotes 
variable-length decoding means for fetching a DCT 
coefficient and other encoded information from a varia- 
ble-length code, 3032 denotes inverse orthogonal trans- 
forming means for applying inverse DCT transformation 
to a DCT coefficient, and 3033 denotes a switching unit 
for switching an output to upside or downside every 
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macroblock in accordance with the encoded information 
showing whether the macroblock is intra-encoded or 
inter-encoded. Symbol 3034 denotes -movement com- 
pensating means for generating a movement compen- 
sating picture by using the last-time decoded picture 5 
and movement encoded information, and adding and 
outputting the outputs of the inverse orthogonal trans- 
forming means 3032. Symbol 3035 denotes execution- 
time measuring means for measuring and outputting the 
execution time until decoding and outputting of a picture 10 
is completed after a variable-length code is input to the 
decoder 303. 

[0287] Symbol 302 denotes estimating means for 
receiving the execution frequency of each element (var- 
iable-length decoding means 3031, inverse orthogonal is 
transforming means 3032, switching unit 3033, or 
movement compensating means 3034) from a data 
string sent from the receiving means 301 and execution 
time from the execution-time measuring means 3035 to 
estimate the execution time of each element. 20 
[0288] To estimate the execution time of each ele- 
ment, it is possible to use the linear regression and 
assume an estimated execution time as a purposed var- 
iable y and the execution frequency of each component 
as an explanatory variable xvi. In this case, it may be 2s 
possible to regard a regression parameter ao>i as the 
execution time of each element Moreover, in the case 
of linear regression, it is necessary to accumulate 
much-enough past data and resultantly, many memo- 
ries are wasted. However, to avoid many memories from. 30 
being wasted. H is also possible to use the estimation of 
an internal-state variable by a Kalman filter. It is possi- 
ble to consider the above case as a case in which an 
observed value is assumed as an execution time, the 
execution time of each element is assumed as an inter- 35 
nal-state variable, and an observation matrix C changes 
every step due to the execution frequency of each ele- 
ment. Symbol 304 denotes frequency reducing means 
for changing the execution frequency of each element 
so as to reduce the execution frequency of fullptxel pre- 40 
diction and increase the execution frequency of halfpixel 
prediction by a corresponding value. The method for 
calculating the corresponding value is shown below. 
[0289] First, the execution frequency and estimated 
execution time of each element are received from the 4s 
estimating means 302 to estimate an execution time. 
When the execution time exceeds the time required to 
receive the data from the receiving means 301 , the exe- 
cution frequency of fullpixel prediction is increased and 
the execution frequency of halfpixel prediction is so 
decreased until the former time does not exceed the lat- 
ter time. Symbol 306 denotes an output terminal for a 
decoded picture. 

[0290] Moreover, there is a case in which the move- 
ment compensating means 3034 is designated so as to 55 
perform halfpixel prediction in accordance with encoded 
information. In this case, when the predetermined exe- 
cution frequency of halfpixel prediction is exceeded, a 



halfpixel movement is rounded to a fullpixel movement 
to execute fullpixel prediction. 
[0291] According to the above-described first and third 
embodiments, the execution time of decoding is esti- 
mated in accordance with the estimated execution time 
of each element and, when the decoding execution time 
may exceed the time (designated time) required to 
receive the data for one sheet, halfpixel prediction hav- 
ing a long execution time is replaced with fullpixel pre- 
diction. Thereby, it is possible to prevent an execution 
time from exceeding a designated time and solve the 
problem (C1) {corresponding to claims 68 and 74). 
[0292] Moreover, a case of regarding the parts of 
indispensable and dispensable processings as two 
groups corresponds to claims 66 and 72 and a case of 
regarding the part of a video as waveform data corre- 
sponds to claims 64 and 70. 

[0293] Furthermore, by using no high-frequency com- 
ponents in the IDCT calculation by a receiver, it is pos- 
sible to reduce the processing time for the IDCT 
calculation. That is, by regarding the calculation of low- 
frequency components as indispensable processing 
and the calculation of high-frequency components as 
dispensable processing in the IDCT calculation, it is 
also possible to reduce the calculation frequency of 
high-frequency components in the IDCT calculation. 
[0294] Figure 41 is a flowchart of the receiving method 
of the fourth embodiment. 

[0295] Because operations of this embodiment are 
similar to those of the third embodiment, corresponding 
elements are added. In step 901. the variable aj for 
expressing the execution time of each element is initial- 
ized (estimating means 302) . In step 902, multiplexed 
data is input and the time required for multiplexing the 
data is measured (receiving means 301) . In step 903, 
the multiplexed data is divided into a variable-length 
code and a data string and output (receiving means 
301). In step 904, each execution frequency is fetched 
from a data string (Figure 2) and it is set to x_i. In step 
905. an actual execution frequency is calculated in 
accordance with the execution time aj of each element 
and each execution frequency x_i (frequency reducing 
means 304). In step 906, measurement of the execution 
time for decoding is started. In step 907, a decoding 
routine to be described later is started. Thereafter, in 
step 908, measurement of the decoding execution time 
is ended (video decoder 303 and execution-time meas- 
uring means 3035). In step 908, the execution time of 
each element is estimated in accordance with the 
decoding execution time in step 908 and the actual exe- 
cution frequency of each element in step 905 to update 
a_i (estimating means 302). The above processing is 
executed every input multiplexed data 
[0296] Moreover, in step 907 for decoding routine, var- 
iable-length decoding is performed in step 910 (varia- 
ble-length decoding means 3031), inverse orthogonal 
transformation is performed in step 91 1 (inverse orthog- 
onal transforming means 3032), and processing is 



26 



k 51 



EP0905 976A1 



52 



branched in step 912 in accordance with the information 
of the intra-/inter-processing fetched through the 
processing in step 91 0 (switching unit 3033). in the case 
of inter-processing, movement compensation is per- 
formed in step 913 (movement compensating means 5 
3034). In step 913, the execution frequency of halfpixel 
prediction is counted in step 913. When the counted 
execution frequency exceeds the actual execution fre- 
quency obtained in step 905, halfpixel prediction is 
replaced with fuilpixel prediction for execution. After the 10 
above processing is applied to every macroblock (step 
914), the routine is ended. 

[0297] According to the above-described second and 
fourth embodiments, the execution time of decoding is 
estimated in accordance with the estimated execution 15 
time of each element and, when the execution time may 
exceed the time required to receive the data for one 
sheet (designated time), harfpixe! prediction having a 
long execution time is replaced with fuilpixel prediction. 
Thereby, it is possible to prevent an execution time from so 
exceeding a designated time and solve the problem 
(C1) (corresponding to claims 67 and 73). 
[0298] Furthermore, a case of regarding the parts of 
dispensable and indispensable processings as two 
groups corresponds to claims 65 and 71 and a case of 2s 
regarding the part of a video as waveform data corre- 
sponds to claims 63 and 69. 

[0299] Figure 36 shows the structure of the receiver of 
the fifth embodiment 

[0300] Most components of this embodiment are the 30 
same as those described for the second embodiment 
However, two added components and one corrected 
component are described below. 
[0301] Symbol 402 denotes estimating means 
obtained by correcting the estimating means 302 35 
described for the second embodiment so as to output 
the execution time of each element obtained as the 
result of estimation separately from an output to fre- 
quency limiting means 304. Symbol 408 denotes trans- 
mitting means for generating the data string shown in 40 
Figure 37 in accordance with the execution time of each 
element and outputting it. When expressing an execu- 
tion time with 16 bits by using microsecond as the unit, 
up to approx. 65 msec can be expressed. Therefore, 
approx. 65 msec will be enough. Symbol 409 denotes 45 
an output terminal for transmitting the data string to 
transmitting means. 

[0302] Moreover, a receiving method corresponding to 
the fifth embodiment can be obtained only by adding a 
step for generating the data string shown in Figure 37 so 
immediately after symbol 808 in Figure 40. 
[0303] Figure 38 shows the structure of the transmitter 
of the sixth embodiment 

[0304] Most components of this embodiment are the 
same as those described for the first embodiment How- ss 
ever, two added components are described below. Sym- 
bol 606 denotes an input terminal for receiving a data 
string output by the receiver of the third embodiment 



and 607 denotes receiving means for receiving the data 
string and outputting the execution time of each ele- 
ment Symbol 608 denotes deciding means for obtain- 
ing the execution frequency of each element and its 
obtaining procedure is described below. First every 
macroblock in a picture is processed by the switching 
unit 1021 to obtain the execution frequency of the 
switching unit 1021 at this point of time. Moreover, it is 
possible to uniquely decide execution frequencies by 
the movement compensating means 1022. orthogonal 
transforming means 1023, and variable-length encod- 
ing means 1024 in accordance with the processing 
result up to this point of time. Therefore, the execution 
time required for decoding at the receiver side is esti- 
mated by using these execution frequencies and the 
execution time sent from the receiving means 607. The 
estimated decoding time is obtained as the total sum of 
the product between the execution time and execution 
frequency of each element every element Moreover, 
when the estimated decoding time is equal to or more 
than the time required to transmit the number of codes 
(e.g. 16 Kbits) to be generated through this picture des- 
ignated by a rate controller or the like (e.g. 250 msec 
when a transmission rate is 64 Kbits/sec), the execution 
frequency of fuilpixel prediction is increased and the 
execution frequency of halfpixel prediction is decreased 
so that the estimated decoding execution time does not 
exceed the time required for transmission. (Because 
fuilpixel prediction has a shorter execution time, it is 
possible to reduce the execution time of fuilpixel predic- 
tion by reducing the frequency of fuilpixel prediction.) 
[0305] Moreover, the video encoder 2102 performs 
various processings in accordance with the execution 
frequency designated by the deciding means 608. For 
example, after the movement compensating means 
1 022 executes halfpixel prediction by the predetermined 
execution frequency of halfpixel prediction, it executes 
only fuilpixel prediction. 

[0306] Furthermore, if is possible to improve the 
selecting method so that halfpixel prediction is uniformly 
dispersed in a picture. For example, it is possible to use 
a method of first obtaining every macroblock requiring 
halfpixel prediction, calculating the product (3) obtained 
by dividing the number of the above macroblocks (e.g. 
12) by the execution frequency of halfpixel prediction 
(e.g 4), and applying halfpixel prediction only to a mac- 
roblock whose sequence from the beginning of the mac- 
roblocks requiring halfpixel prediction is divided by the 
above product without a remainder (0. 3, 6, or 9). 
[0307] According to the above-described fifth and 
sixth embodiments , the execution time of each esti- 
mated element is transmitted to the transmitting side, 
the execution time of decoding is estimated at the trans- 
mitting side, and halfpixel prediction having a long exe- 
cution time is replaced with fuilpixel prediction so that 
the estimated decoding execution time does not exceed 
the time (designated time) probably required to receive 
the data for one sheet. Thereby, the information for half- 
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pixel prediction among the sent encoded information is 
not disused and thereby, it is possible to prevent an exe- 
cution time from exceeding a designated time and solve 
the problem (C2) (corresponding to claims 76 and 78). 
[0308] Moreover, in the case of dispensable process- 
ing, it is possible to divide inter-macrobiock encoding 
into such three movement compensations as normal 
movement compensation, 8x8 movement compensa- 
tion, and overlap movement compensation. 
[0309] Figure 42 is a flowchart of the transmitting 
method of the seventh embodiment. 
[0310] Because operations of this embodiment are 
similar to those of the sixth embodiment, corresponding 
elements are added. In step 1001 , the initial value of the 
execution time of each processing is set. A picture is 
input (input terminal 2101) in step 801 and it is divided 
into macroblocks in step 802. In step 1002, it is decided 
whether to intra-encode or inter-encode every macrob- 
lock (switching unit 1021). Resultantly, the execution 
frequency of each processing from step 1005 to step 
806 is known. Therefore, in step 1003, an actual execu- 
tion frequency is calculated in accordance with the 
above execution frequency and the execution time of 
each processing (deciding means 608). 
[031 1] Hereafter, the processings from step 1005 to 
step 806. are repeated until the processing for every 
macroblock is completed in accordance with the condi- 
tional branch in step 807. 

[0312] Moreover, when each processing is executed, 
a corresponding variable is incremented by 1 so that the 
processing frequencies from step 1005 to step 806 can 
be recorded in a specific variable. First, in step 1005. 
branching is performed in accordance with the decision 
result in step 1002 (switching unit 1021). In the case of 
inter-encoding, movement compensation is performed 
in step 804 (movement compensating means 1022). In 
this case, the frequency of halfpixei prediction is 
counted. When the counted frequency exceeds the 
actual frequency obtained in step 1003, fullpixel predic- 
tion is executed instead without executing halfpixei pre- 
diction. Thereafter, in steps 805 and 806, DCT 
transformation and variable-length encoding are per- 
formed (orthogonal transforming means 1023 and vari- 
able-length encoding means 1024). When the 
processing for every macroblock is completed, (in the 
case of Yes in step 807), the variable showing the exe- 
cution frequency corresponding to each processing is 
read in step 808, the data string shown in Figure 2 is 
generated, and the data string and a code are multi- 
plexed and output. In step 1004, the data string is 
received and the execution time of each processing is 
fetched from the data string and set. 
[0313] Processings from step 801 to step 1004 are 
repeatedly executed as long as pictures are input. 
[031 4] According to the paragraph beginning with the 
final "Moreover" of the descriptive portion of the fifth 
embodiment and the seventh embodiment, the esti- 
mated execution time of each element is transmitted to 
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the transmitting side, the execution time of decoding is 
estimated at the transmitting side, and hatfpixel predic- 
tion having a long execution time is replaced with full- 
pixel prediction so that the . estimated decoding 
5 execution time does not exceed the time (designated 
time) probably required to receive the data for one 
sheet. Thereby, the information for halfpixei prediction 
among the sent encoded information is not disused and 
it is possible to prevent the execution time from exceed- 
to ing the designated time and solve the problem (C2) 
(corresponding to claims 75 and 77). 
[031 5] Figure 39 shows the structure of the transmit- 
ting apparatus of the eighth embodiment of the present 
invention. 

is [0316] Most components of this embodiment are the 
same as those described for the first embodiment. 
Therefore, four added components are described 
below. 

[0317] Symbol 7010 denotes execution-time measur- 
20 ing means for measuring the execution time until encod- 
ing and outputting of a picture are completed after the 
picture is input to an encoder 2102 and outputting the 
measured execution time. Symbol 706 denotes estimat- 
ing means for receiving execution frequencies of ele- 
25 ments (switching unit 1021, movement compensating 
means 1022, orthogonal transforming means 1023, and 
variable-length decoding means 1024) of a data string 
from counting means 2103 and the execution time from 
the execution-time measuring means 7010 and estimat- 
ed ing the execution time of each element It is possible to 
use an estimating method same as that described for 
the estimating means 302 of the second embodiment. 
Symbol 707 denotes an input terminal for inputting a 
frame rate value sent from a user and 708 denotes 
35 deciding means for obtaining the execution frequency of 
each element. The obtaining procedure is descrtoed 
below. 

[0318] First, every macroblock in a picture is proc- 
essed by the switching unit 1021 to obtain the execution 

40 frequency of the switching unit 1021 at this point of time. 
Thereafter, it is possible to uniquely decide execution 
frequencies by the movement compensating means 
1022, orthogonal transforming means 1023, and varia- 
ble-length encoding means 1 024 in accordance with the 

45 processing result up to this point of time. Then, the total 
sum of products between the execution frequency and 
the estimated execution time of each element sent from 
the estimating means 706 is obtained every element to 
calculate an estimated encoding time. Then, when the 

so estimated encoring time is equal to or longer than the 
time usable for encoding of a sheet of picture obtained 
from the inverse number of the frame rate sent from 
symbol 707, the execution frequency of fullpixel predic- 
tion is increased and that of halfpixei prediction is 

55 decreased. 

[0319] By repeating the above change of execution 
frequencies and calculation of the estimated encoding 
time until the estimated encoding time becomes equal 
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to or shorter than the usable time, each execution fre- 
quency is decided. 

[0320] Moreover, the video encoder 2102 performs 
various processings in accordance with the execution 
frequency designated by the deciding means 608. For 5 
example, after the movement compensating means 
1022 executes halfpixel prediction by the predetermined 
execution frequency of halfpixel prediction, it executes 
only fullpixel prediction. 

[0321] Furthermore, it is also possible to improve a 10 
selecting method so that halfpixel prediction is uniformly 
dispersed in a picture For example, it is possible to use 
a method of obtaining every macroblcck requiring half- 
pixel prediction, calculating the product (3) obtained by 
dividing the number of macroblocks requiring halfpixel 75 
prediction (e.g. 12) by the execution frequency of half- 
pixel prediction (e.g. 4), and applying halfpixel predic- 
tion only to a macroblock whose sequence from the 
beginning of the macroblocks requiring halfpixel predic- 
tion is divided by the product without remainder (0, 3, 6, 20 
or 9). 

[0322] The above eighth embodiment makes it possi- 
ble to solve the problem (C3) by estimating the execu- 
tion time of each processing, estimating an execution 
time required for encoding in accordance with the esti- 25 
mated execution time, and deciding an execution fre- 
quency so that the estimated encoding time becomes 
equal to or shorter than the time usable for encoding of 
a picture determined in accordance with a frame rate 
(corresponding to claim 80). 30 
[0323] Moreover, because the movement compensat- 
ing means 1022 detects a movement vector, there is a 
full-search movement-vector detecting method for 
detecting a vector for minimizing SAD (sum of absolute 
values of differences every pixel) among vectors in a 35 
range of 15 horizontal and vertical pixels. Furthermore, 
there is a three-step movement-vector detecting 
method (described in annex of H.261). The three-step 
movement-vector detecting method executes the 
processing of selecting nine points uniformly arranged 40 
in the above retrieval range to select a point having a 
minimum SAD and then, selecting nine points again in a 
narrow range close to the above point to select a point 
having a minimum SAD one more time: 
[0324] It is also possible to properly decrease the exe- 45 
cution frequency of the full-search movement-vector 
detecting method and properly increase the execution 
frequency of the three-step movement-vector detecting 
method by regarding these two methods as a dispensa- 
ble processing method and estimating the execution so 
time of each of the two methods, estimating an execu- 
tion time required for encoding in accordance with the 
estimated execution time so that the estimated execu- 
tion time becomes equal to or shorter than the time des- 
ignated by a user. 55 
[0325] Moreover, it is possible to use a movement- 
vector detecting method using a fixed retrieval fre- 
quency and further simplifying the processing or a 
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movement-vector detecting method of returning only 
the movement vector (0, 0) as a result together with the 
three-step movement-vector detecting method. 
[0326] Figure 43 is a flowchart of the transmitting 
method of the ninth embodiment. 
[0327] Because operations of this embodiment are 
similar to those of the eighth embodiment, correspond- 
ing elements are added. For the detailed operation in 
each flow, refer to the description of corresponding ele- 
ments. 

[0328] Moreover, because this embodiment is almost 
the same as the second embodiment, only different 
points are explained below. 

[0329] In step 1 1 01 , the initial value of the execution 
time of each processing is set to a variable aj. Moreo- 
ver, in step 1102, a frame rate is input (input terminal 
707). In step 1103, an actual execution frequency is 
decided in accordance with the frame rate and the exe- 
cution time a j of each processing in step 1 102 and the 
execution frequency of each processing obtained from 
the intraVinter-processing decision result in step 1002 
(deciding means 708). In steps 1 105 and 1 1 06, the exe- 
cution time of encoding is measured. In step 1104, the 
execution time of each processing is estimated in 
accordance with the execution time obtained in step 
1106 and the actual execution frequency of each 
processing to update the variable aj (estimating means 
706). 

[0330] According to the above-described ninth 
embodiment, the execution time of each processing is 
estimated and an execution time required for encoding 
is previously measured in accordance with the esti- 
mated execution time. Thus, it is possible to solve the 
problem (C3) by deciding an actual execution frequency 
so that the estimated encoding time becomes the time 
usable for the encoding of a picture determined in 
accordance with a frame rate or shorter (corresponding 
to claim 79). 

[0331 ] In the case of the second embodiment, it is also 
possible to add a two-byte region immediately after the 
start code shown in Figure 2 when the data string is 
generated in step 808 and add the binary notation of a 
code length to the region. 

[0332] Moreover, in the case of the fourth embodi- 
ment, it is also possible to extract a code length from the 
two-byte region when multiplexed data is input in step 
902 and use the code transmission time obtained from 
the code length and the code transmission rate for the 
execution frequency calculation in step 905 (the execu- 
tion frequency of halfpixel prediction is decreased so as 
not to exceed the code transmission time). This corre- 
sponds to claims 81 and 83. 

[0333] Furthermore, in the case of the first embodi- 
ment, it is also possible to add a two-byte region imme- 
diately after the start code shown in Figure 2 when a 
data string is generated in step 2104 and add the binary 
notation of a code length to the region. 
[0334] Furthermore, in the case of the third embodi- 
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ment, it is also possible to extract a code length from the 
two-byte region when multiplexed data is input in step 
301 and use a code transmission time obtained from the 
code length and the code transmission rate for the exe- 
cution frequency calculation in step 304 (the execution 5 
frequency of haHpixel prediction is decreased so as not 
to exceed the code transmission time). This corre- 
sponds to claims 82 and 84. 

[0335] Furthermore, in the case of the fourth embodi- 
ment, an actual execution frequency of halfpixel predic- 10 
tion is recorded immediately after step 909 to calculate 
a maximum value. When the maximum value is equal to 
or less than a small-enough value (e.g. 2 or 3), it is also 
possible to generate a data string (data string compris- 
ing a specific bit pattern) showing that halfpixel predic- 75 
tion is not used and transmit the generated data string. 
Furthermore, in the case of the second embodiment it 
is confirmed whether the data string is received immedi- 
ately after step 808 and when the data string showing 
that halfpixel prediction is not used is received, it is also 20 
possible to make movement compensation processing 
always serve as fullpixel prediction in step 808. This cor- 
responds to claims 93 and 91 . 
[0336] Furthermore, the above concept can be 
applied to cases other than movement compensation. 25 
For example, it is possible to reduce the DCT calcula- 
tion time by using no high-frequency component for 
DCT calculation. That is, in the case of a receiving 
method, when the rate of the IDCT-calculation execution 
time to the entire execution time exceeds a certain 30 
value, a data string showing that the rate exceeds a cer- 
tain value is transmitted to the transmitting side. When 
the transmitting side receives the data string, it is also 
possible to calculate only low-frequency components 
through the DCT calculation and decrease all high-fre- 35 
quency components to zero. This corresponds to claim 
89. 

[0337] Furthermore, though the embodiment is 
described above by using a picture, it is possible to 
apply each of the above methods to audio instead of ao 
video. This corresponds to claims 85 and 87. 
[0338] Furthermore, in the case of the third embodi- 
ment, an actual execution frequency of halfpixel predic- 
tion is recorded in step 3034 to calculate a maximum 
execution frequency. Then, when the maximum value is 45 
a small-enough value or less (e.g. 2 or 3), it is possible 
to generate and transmit a data string showing that half- 
pixel prediction is not used (data string comprising a 
specific bit pattern). Furthermore, in the case of the first 
embodiment, when receiving a data string showing that so 
halfpixel prediction is not used, it is possible to make the 
movement compensation processing in step 1022 
always serve as fullpixel prediction. This corresponds to 
claims 94 and 92. 

[0339] Furthermore, the above concept can be 55 
applied to cases other than movement compensation. 
For example, by using no high-frequency component for 
DCT calculation, it is possible to reduce the DCT calcu- 



lation processing time. That is, in the case of a receiving 
method, when the rate of IDCT-calculation execution 
time to the entire execution time exceeds a certain 
value, a data string showing that the rate exceeds a cer- 
tain value is transmitted to the transmitting side. 
[0340] When the transmitting side receives the data 
string, it is possible to calculate only low-frequency 
components through the DCT calculation and reduce all 
high-frequency components to zero. This corresponds 
to claim 90. 

[0341] Furthermore, though the embodiment is 
described above by using a picture, it is also possible to 
apply the above method to audio instead of picture. This 
corresponds to claims 86 and 88. 
[0342] As described above, according to claims 68 
and 74 (e.g. first and third embodiments), the execution 
time of decoding is estimated in accordance with the 
estimated execution time of each element and, when 
the estimated decoding execution time may exceed the 
time (designated time) required to receive the data for 
one sheet, halfpixel prediction having a long execution 
time is replaced with fullpixel prediction. Thereby, it is 
possible to prevent the execution time from exceeding 
the designated time and solve the problem (C1). 
[0343] Furthermore, according to claims 75 and 77 
(e.g. fifth and seventh embodiments), the estimated 
execution time of each element is transmitted to the 
transmitting side, the execution time of decoding is esti- 
mated at the transmitting side, and halfpixel prediction 
having a long execution time is replaced with fullpixel 
prediction so that the estimated decoding time does not 
exceed the time (designated time) probably required to 
receive the data for one sheet. Thereby, the information 
for halfpixel prediction in the sent encoded information 
is not disused and it is possible to prevent the execution 
time from exceeding the designated time and solve the 
problem (C2). 

[0344] Furthermore, according to claim 79 (e.g. ninth 
embodiment), ft is possible to solve the problem (C3) by 
estimating the execution time of each processing, more- 
over estimating the execution time required for encoding 
in accordance with the estimated execution time, and 
deciding an executing frequency so that the estimated 
encoding time becomes equal to or less than the time 
usable for encoding of a picture decided in accordance 
with a frame rate. 

[0345] Thus, the present invention makes it possible 
to realize a function (COD: Computational Graceful 
Degradation) for slowly degrading quality even if a cal- 
culated load increases and thereby, a very large advan- 
tage can be obtained. 

[0346] Moreover, it is possible to perform operations 
same as described above by a computer by using a 
recording medium such as a magnetic recording 
medium or optical recording medium in which a pro- 
gram for making the computer execute all or part (or 
operations of each means) of the each steps (or each 
means) described in any one of the above-described 
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embodiments. 
Industrial Applicability 

[0347] As described above, the present invention s 
makes it possible to change information frames corre- 
spondingly to the situation, purpose, or transmission 
line by dynamically deciding the frames of data control 
information, transmission control information, and con- 
trol information used for transmitting and receiving ter- 10 
minals. Moreover, it is easy to handle a plurality of video 
streams or a plurality of audio streams and mainly 
reproducing an important scene cut synchronously with 
audio by reflecting the intention of an editor. Further- 
more, it is possible to prevent an execution time from 15 
exceeding a designated time by estimating the execu- 
tion time of decoding in accordance with the execution 
time of each estimated element and replacing harfpixel 
prediction having a long execution time with fullpixel 
prediction when the estimated decoding execution time 20 
may exceed the time (designated time) required to 
receive the data for one sheet 

Claims 

25 

1. An audio-video transmitting apparatus comprising 
transmitting means for transmitting the content con- 
cerned with a transmitting method and/or the struc- 
ture of data to be transmitted or an identifier 
showing the content as transmission format infor- 30 
mation through a transmission line same as that of 

the data to be transmitted or a transmission line dif- 
ferent from the data transmission line; wherein 

said data to be transmitted is video data and/or 35 
audio data. 

2. The audio-video transmitting apparatus according 
to claim 1, wherein said transmission format infor- 
mation is included in at least one of data control 40 
information added to said data to control said data, 
transmission control information added to said data 

to transmit said data, and information for controlling 
the processing of the terminal side. 

45 

3. The audio-video transmitting apparatus according 
to claim 2, wherein at least one of said data control 
information, transmission control information, and 
information for controlling the processing of said 
terminal side in dynamically changed. so 

4. The audio-video transmitting apparatus according 
to claim 3, wherein 

said data is divided into a plurality of packets, 55 
and 

said data control information or said transmis- 
sion control information is added not only to the 




head packet of said divided packets but also to 
a middle packet of them. 

5. The audio-video transmitting apparatus according 
to claim 1 , wherein an identifier showing whether to 
use timing information concerned with said data as 
information showing the reproducing time of said 
data is included in said transmission format infor- 
mation. 

6. The audio-video transmitting apparatus according 
to claim 1, wherein said transmission format infor- 
mation is the structural information of said data and 
a signal which is output from a receiving apparatus 
receiving the transmitted structural information of 
said data and which can be received is confirmed 
and thereafter, said transmitting means transmits 
corresponding data to said receiving apparatus. 

7. The audio-video transmitting apparatus according 
to claim 1, wherein said transmission format infor- 
mation include (1) an identifier for identifying a pro- 
gram or data to be used by a receiving apparatus 
later and (2) at least one of a flag, counter, and 
timer as information for knowing the point of time in 
which said program or data is used or the term of 
validity for using said program or data. 

8. The audio-video transmitting apparatus according 
to claim 7, wherein said point of time in which said 
program or data is used is transmitted as transmis- 
sion control information by using a transmission 
serial number for identifying a transmission 
sequence or as information to be transmitted by a 
packet different from that of data to control terminal- 
side processing. 

9. The audio-video transmitting apparatus according 
to claim 2 or 3, wherein 

storing means for storing a plurality of contents 
concerned with said transmitting method 
and/or said structure of data to be transmitted 
and a plurality of its identifiers are included, 
and 

said identifier is included in at least one of said 
data control information, transmission control 
information, and information for controlling ter- 
minal-side processing as said transmission for- 
mat information. 

10. The audio-video transmitting apparatus according 
to claim 2 or 3, wherein storing means for storing a 
plurality of contents concerned with said transmit- 
ting method and/or said structure of data to be 
transmitted are included, and 

said contents are included in at least one of 
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said data control information, transmission 
control information, and information for control- 
ling terminal-side processing as said transmis- 
sion format information. 

5 

11. The audio-video transmitting apparatus according 
to claim 1 , 2, or 3, wherein a default identifier show- 
ing whether to change the contents concerned with 
said transmitting method and/or structure of data to 

be transmitted is added. 10 

12. The audio-video transmitting apparatus according 
to claim 9, 10, or 1 1, wherein said identifier or said 
default identifier is added to a predetermined fixed- 
length region of information to be transmitted or 15 
said predetermined position. 

13. An audio-video receiving apparatus comprising: 




arranging said information without multiplexing it 
before said data and/or control information or trans- 
mit said multiplexing control information through a 
transmission line different from the transmission 
line for transmitting said data and/or control infor- 
mation. 

17. An audio-video receiving apparatus comprising: 

receiving means for receiving said multiplexing 
control information transmitted from the audio- 
video transmitting apparatus of claim 15 and 
said multiplexed data and/or control informa- 
tion; and 

separating means for separating said multi- 
plexed data and/or control information in 
accordance with said multiplexing control infor- 
mation. 



receiving means for receiving said transmis- 20 
sion format information transmitted from the 
audio-video transmitting apparatus of any one 
of claims 1 to 12; and 

transmitted-information interpreting means for 
interpreting said received transmission-format 2s 
information. 

14. The audio-video receiving apparatus according to 
claim 13, wherein 

30 

storing means for storing a plurality of contents 
concerned with said transmitting method 
and/or said structure of data to be transmitted 
and a plurality of its identifiers are included, 
and 35 
the contents stored in said storing means are 
used to interpret said transmission format infor- 
mation. 

1 5. An audio-video transmitting apparatus comprising: 40 

information multiplexing means for controlling 
start and end of multiplexing the information for 
a plurality of logical transmission lines for trans- 
mitting data and/or control information is as 
included; wherein, 

not only said data and/or control information 
multiplexed by said information multiplexing 
means but also control contents concerned 
with start and end of said multiplexing by said so 
information multiplexing means are transmitted 
as multiplexing control information, and 
said data includes video data and/or audio 
data. 

55 

16. The audio-video transmitting apparatus according 
to claim 15, wherein it is possible to select whether 
to transmit said multiplexing control information by 



18. An audio-video receiving apparatus comprising: 

main looKing-iistening means for looking at and 
listening to a broadcast program; and 
auxiliary looking-listening means for cyclically 
detecting the state of a broadcast program 
other than the broadcast program looked and 
listened through said main looking-listening 
means: wherein 

said detection is performed so that a program 
and/or data necessary when said broadcast 
program looked and listened through said main 
looking-listening means is switched to other 
broadcast program can be smoothly proc- 
essed, and 

said data includes video data and/or audio 
data. 

19. The audio-video transmitting apparatus according 
to claim 1, wherein priority values can be changed 
in accordance with the situation by transmitting the 
offset value of information showing the priority for 
processing of said data. 

20. An audio-video receiving apparatus comprising: 

receiving means for receiving encoded infor- 
mation to which the information concerned with 
the priority for processing under an overload 
state is previously added; and 
priority deciding means for deciding a threshold 
serving as a criterion for selecting whether to 
process an object in said information received 
by said receiving means; wherein 
the tinting for outputting said received informa- 
tion is compared with the elapsed time after 
start of processing or the timing for decoding 
said received information is compared with the 
elapsed time after start of processing to 
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change said threshold in accordance with the 
comparison result, and 

video data and/or audio data are or is included 
as said encoding object. 

5 

21. The audio-video receiving apparatus according to 
claim 20, wherein 

retransmission-request-priority deciding 
means for deciding a threshold serving as a cri- 10 
terion for selecting whether to request retrans- 
mission of some of said information not 
received because it is lost under transmission 
when it is necessary to retransmit said informa- 
tion is included, and 15 
said decided threshold is decided in accord- 
ance with at least one of the priority controlled 
by said priority deciding means, retransmission 
frequency, lost factor of information, insertion 
interval between in-frame-encoded frames, 20 
and grading of priority. 

22. An audio-video transmitting apparatus comprising: 

retransmission-priority deciding means for 25 
deciding a threshold serving as a criterion for 
selecting whether to request retransmission of 
some of said information not received because 
it is lost under transmission when retransmis- 
sion of said unreceived information is 30 
requested is included, wherein 
said decided threshold is decided in accord- 
ance with at least one of the priority controlled 
by the priority deciding means of said audio- 
video receiving apparatus of claim 20, retrans- 35 
mission frequency, lost factor of information, 
insertion interval between in-frame-encoded 
frames, and grading of priority. 

23. An audio-video transmitting apparatus for transmit- 40 
ting said encoded information by using the priority 
added to said encoded information and thereby 
thinning it when (1) an actual transfer rate exceeds 
the target transfer rate of information for a video or 
audio or (2) it is decided that writing of said 45 
encoded information into a transmitting buffer is 
delayed as the result of comparing the elapsed time 
after start of transmission with a period to be 
decoded or output added to said encoded informa- 
tion. 50 

24. A data processing method comprising the steps of: 

inputting a data series including (1) time-series 
data for audio or video, (2) an inter-time-series- 55 
data priority showing the priority of the 
processing between said time-series-data val- 
ues, and (3) a plurality of in-time-series-data 




priorities for dividing said time-series data 
value to show the processing priority between 
divided data values; and 
performing processing by using said inter-time- 
series-data priority and said in-time-series- 
data priority together when pluralities of said 
time-series-data values are simultaneously 
present. 

25. A data processing apparatus comprising: 

receiving means for receiving a data series 
including (1) time-series data for audio or 
video, (2) an inter-time-series-data priority 
showing the priority of the processing between 
said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said 
time-series data value to show the processing 
priority between divided data values; and 
data processing means for performing 
processing by using said inter-time-series-data 
priority and said in-time-series-data priority 
together when pluralities of said time-series- 
data values are simultaneously present. 

26. A data processing method comprising the steps of: 

inputting a data series including (1) time-series 
data for audio or video, (2) an inter-time-series- 
data priority showing the priority of the 
processing between said time-series-data val- 
ues, and (3) a plurality of in-time-series-data 
priorities for dividing said time-series data 
value to show the processing priority between 
divided data values; and 
distrbuting throughput to each of said time- 
series-data values in accordance with said 
inter-time-series-data priority and moreover, 
adaptively deteriorating the processing quality 
of the divided data in said time-series data in 
accordance with said in-time-series-data prior- 
ity so that each of said time-series-data values 
is kept within said distributed throughput. 

27. A data processing apparatus comprising: 

receiving means for receiving a data series 
including (1) time-series data for audio or 
video, (2) an inter-time-series-data priority 
showing the priority of the processing between 
said time-series-data values, and (3) a plurality 
of in-time-series-data priorities for dividing said 
time-series data value to show the processing 
priority between divided data values; and 
data processing means for distributing through- 
put to each of said time-series-data values in 
accordance with said inter-time-series-data pri- 
ority and moreover, adaptively deteriorating the 
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processing quality of the divided data in said 
time-series data in accordance with said in- 
time-series-data priority so that each of said 
time-series-data values is kept within said dis- 
tributed throughput. 5 

28. A data processing method characterized by, when 
an in-time-series-data priority for a video is added 
every frame of said video and said video for each 
frame is divided into a plurality of packets, 10 

adding said in-time-series-data priority only to 
the header portion of a packet for transmitting 
the head portion of a frame of said video 
accessible as independent information. 15 

29. A data processing apparatus characterized by, 
when an in-time-series-data priority for a video is 
added every frame of said video and said video for 
each frame is divided into a plurality of packets, 20 

adding said in-time-series-data priority only to 
the header portion of a packet for transmitting 
the head portion of a frame of said video 
accessible as independent information. 25 

30. The data processing method according to any one 
of claims 24, 26, and 28, wherein said in-time- 
series-data priority is described in the header of a 
packet to perform priority processing. 30 

31. The data processing apparatus according to any 
one of claims 25. 27, and 29, wherein said in-time- 
series-data priority is described in the header of a 
packet to perform priority processing. 35 

32. The data processing method according to any one 
of claims 24, 26, and 28, wherein the range of a 
value capable of expressing said in-time-series- 
data priority is made variable to perform priority 40 
processing. 

33. The data processing apparatus according to any 
one of claims 25, 27, and 29, wherein the range of 

a value capable of expressing said in-time-series- 45 
data priority is made variable to perform priority 
processing. 
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35. A data processing apparatus characterized by: 



34. A data processing method comprising the steps of: 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between said time-series data values; and 
processing priorities by using said inter-time- 
series-data priority as the value of a relative or 
absolute priority. 
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inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between said time-series data values; and 
processing priorities by using said inter-time- 
series-data priority as the value of a relative or 
absolute priority. 

36. A data processing method comprising the steps of: 

classifying time-series data values for audio or 
video; 

inputting a data series including said time- 
series data and a plurality of in-time-series- 
data priorities showing the processing priority 
between said classified data values; and 
processing priorities by using said in-time- 
series-data priority as the value of a relative or 
absolute priority. 

37. A data processing apparatus characterized by: 

classifying time-series data values for audio or 
video; 

inputting a data series including said time- 
series data and a plurality of in-time-series- 
data priorities showing the processing priority 
between said classified data values; and 
processing priorities by using said in-time- 
series-data priority as the value of a relative or 
absolute priority. 

38. A data processing method comprising the steps of: 

classifying time-series data values for audio or 
video; 

encoding said classified data values; 
inputting a data series describing an in-time- 
series-data priority serving as the value of an 
absolute priority in said encoded information 
and a in-time-series-data priority serving as the 
value of a relative priority in the header portion 
of a packet constituted with said encoded infor- 
mation; and 
processing priorities. 

39. A data processing apparatus characterized by: 

classifying time-series data values for audio or 
video; 

encoding said classified data values; 
inputting a data series describing an in-time- 
series-data priority serving as the value of an 
absolute priority in said encoded information 
and a in-time-series-data priority serving as the 
value of a relative priority in the header portion 
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of a packet constituted with said encoded infor- 
mation; and 
processing priorities. 

40. A data processing method comprising the steps of: 5 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between time series data values; and 10 
processing priorities by relating one said inter- 
time-series-data priority or more to a TCP/IP 
logical channel. 

41. A data processing apparatus characterized by: 15 

inputting a data series including time-series 
data for audio or video and an inter-time- 
series-data priority showing the processing pri- 
ority between time series data values; and 20 
processing priorities by relating one said inter- 
time-series-data priority or more to a TCP/IP 
logical channel. 

42. The data processing method according to claim 34 2s 
or 36, wherein 

said priority processing is performed (1) by 
using said inter-time-series-data priority as the 
value of a relative priority when accumulating 30 
and using said inter-time-series-data priority 
and (2) by using said inter-time-series-data pri- 
ority as the value of an absolute priority when 
transmitting said data. 

35 

43. The data processing apparatus according to claim 
35 or 37, wherein 

said priority processing is performed (1) by 
expressing said inter-time-series-data priority 40 
as the value of a relative priority when accumu- 
lating and using said inter-time-series-data pri- 
ority and (2) by expressing said inter-time- 
series-data priority as the value of an absolute 
priority when transmitting said inter-time- 45 
series-data priority. 

44. The data processing method according to claim 34 
or 36, wherein 

50 

an identifier classifies whether to express the 
value of said priority as a relative value or an 
absolute value. 

45. The data processing apparatus according to claim 55 
35 or 37, wherein 

an identifier classifies whether to express the 




value of said priority as a relative value or an 
absolute value. 

46. A data processing method comprising the steps of: 

when one time-series data includes a plurality 
of sub-time-series data values, describing the 
relation between said sub-time-series data val- 
ues and thereby defining a method for process- 
ing said sub-time-series data to perform priority 
processing. 

47. A data processing apparatus characterized by 
when one time-series data includes a plurality of 
sub-time-series data values, describing the relation 
between said sub-time-series data values and 
thereby defining a method for processing said sub- 
time-series data to perform priority processing. 

48. The data processing method according to any one 
of claims 34, 36, and 46, wherein a packet-consti- 
tuting method is decided in accordance with any 
one of said inter-time-series-data priority, in-time- 
series-data priority, and relational description 
between said time-series data values. 

49. The data processing apparatus according to any 
one of claims 35, 37, and 47, wherein a packet-con- 
stituting method is decided in accordance with any 
one of said inter-time-series-data priority, in-time- 
series-data priority, and relational description 
between said time-series data values. 

50. A data processing method characterized by relating 
the sliced structure of a video to the structure of a 
packet and thereby, making a re-sync marker for 
resynchronization unnecessary. 

51. A data processing apparatus characterized by 
relating the sliced structure of a video to the struc- 
ture of a packet and thereby, making a re-sync 
marker for resynchronization unnecessary. 

52. A data processing apparatus characterized by 
transmitting a method for relating time-series data 
for audio or video to a packet together with control 
information or said time-series data and thereby, 
defining relating of said time-series data to said 
packet. 

53. The data processing method according to claim 48, 
wherein high error protection is applied to a packet 
including the information for said high in-time- 
series-data priority or inter-time-series-data priority. 

54. The data processing apparatus according to claim 
49, wherein high error protection is applied to a 
packet including the information for said high in- 
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time-series-data priority or inter-time-series-data 
priority. 

55. The data processing method according to claim 34 
or 36, wherein s 



56. The data processing apparatus according to claim 

35 or 37, wherein is 

a priority added to a packet is used as a packet 
priority, and 

said priority processing is performed by relating 
at least either of the values of said in-time- 20 
series-data priority and said inter-time-series- 
data priority to said packet priority. 

57. The data processing method according to claim 34 

or 36, wherein 25 

said priority processing is performed by assign- 
ing a value lower than a character or control 
information to said time-series data as the 
value of said in-time-series-data priority or said 30 
inter-time-series-data priority. 

58. The data processing apparatus according to claim 
35 or 37, wherein said priority processing is per- 
formed by assigning a value lower than a character 35 
or control information to said time-series data as 
the value of said in-time-series-data priority or said 
inter-time-series-data priority. 

59. A data processing method comprising the steps of: 40 

successively inputting classified time-series 
data and its priority information; and 

(1) when the information for said classified 45 
time-series data is damaged, performing 
retransmission request processing in order 
to request retransmission of said damaged 
data and (2) when said classified time- 
series data is continuously or frequently so 
lost, applying said retransmission request 
processing only to high-priority data. 

60. A data processing apparatus characterized by, suc- 
cessively inputting classified time-series data and 55 
its priority information; and 

(1) when the information for said classified 



time-series data is damaged, performing 
retransmission request processing in order to 
request retransmission of said damaged data 
and, (2) when said classified time-series data 
is continuously or frequently lost, applying said 
retransmission request processing only to 
high-priority data. 

61 . A data processing method comprising the step of: 

successively inputting classified time-series 
data and its priority information; and 
preferentially transmitting said high-priority 
data in accordance with the amount of said 
classified time-series data to be transmitted. 

62. A data processing apparatus characterized by: 

successively inputting classified time-series 
data and its priority information; and 
preferentially transmitting said high-priority 
data in accordance with the amount of said 
classified time-series data to be transmitted. 

63. A waveform data transmitting method comprising 
the steps of: 

(a) dividing a plurality of decoding units consti- 
tuting the waveform-data decoding process 
into a plurality of groups in accordance with the 
significance for maintaining quality and count- 
ing the execution frequency of an encoding unit 
corresponding to the decoding unit belonging 
to each group; 

(b) receiving said counted result and transform- 
ing said result into a data string when encoding 
of waveform data for a predetermined period is 
completed; and 

(c) outputting a code which is a waveform-data 
encoding result and said data string and trans- 
mitting the execution frequency of each 
processing unit every a plurality of groups to 
the receiving apparatus. 

64. A waveform data transmitting apparatus compris- 
ing: 

(a) counting means for dividing a plurality of 
decoding units constituting the waveform-data 
decoding process into a plurality of groups in 
accordance with the significance for maintain- 
ing quality and counting the execution fre- 
quency of an encoding unit corresponding to 
the decoding unit belonging to each group; 

(b) transforming means for receiving said 
counted result and transforming said result into 
a data string when encoding of waveform data 
for a predetermined period is completed; and 



a priority added to a packet is used as a packet 
priority, and 

said priority processing is performed by relating 
at least either of the values of said in-time- 10 
series-data priority and said inter-time-series- 
data priority to said packet priority. 
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(c) transmitting means for oirtputting a code 
which is a waveform-data encoding result and 
said data string; wherein _ 
the execution frequency of each processing 
unit is transmitted to the receiving apparatus 5 
every a plurality of groups. 

65. The waveform data transmitting method according 
to claim 63, wherein 

10 

pluralities of decoding units constituting a plu- 
rality of waveform-data decoding processes 
are divided into at least one indispensable 
processing or more and at least one dispensa- 
ble processing or more (when this processing 15 
is omitted, waveforms are deteriorated but 
waveforms can be decoded), the execution fre- 
quency of said indispensable processing and 
that of dispensable processing are counted, 
and the execution f requ enct es of said indispen- 20 
sable and dispensable processings for each 
processing unit are transmitted to said receiv- 
ing apparatus. 

66. The waveform data transmitting apparatus accord- 2s 
ing to claim 64, wherein counting means for dividing 

a plurality of decoding units constituting a plurality 
of waveform-data decoding processes into at least 
one indispensable processing or more and at least 
one dispensable processing or more (when this 30 
processing is omitted, waveforms are deteriorated 
but waveforms can be decoded} and counting the 
execution frequency of said indispensable process- 
ing and that of dispensable processing is included 
and the execution frequencies of said indispensa- 35 
ble and dispensable processings for each process- 
ing unit are transmitted to said receiving apparatus. 

67. The video waveform data transmitting method 
according to claim 63. wherein a video is input as 40 
said waveform data. 

68. The video waveform data transmitting apparatus 
according to claim 64, wherein a video is input as 
said waveform data. 45 

69. A waveform data receiving method comprising the 
steps of: 

(a) receiving a data string including the code of so 
waveform data and the execution frequency of 
each decoding unit grouped in accordance with 

the significance for maintaining the quality of 
the waveform data decoded from said code 
and oirtputting said code and said execution 55 
frequency; 

(b) estimating the execution time of each group 
in accordance with the processing time until 
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obtaining a waveform after decoding said code 
and each of said execution frequencies 
obtained from said data string; and 
(c) estimating the processing time required to 
decode a waveform by using the execution fre- 
quency and said execution time, calculating the 
reduced number of execution frequencies of # 
groups in which said processing time does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
referred to as designated time) in accordance 
with each execution time output by said receiv- 
ing means and each execution time output by 
said estimating means, estimating the time 
required for decoding, and reducing the execu- 
tion frequency of each group so as to complete 
decoding within said designated time. 

70. A waveform data receiving apparatus comprising: 

(a) receiving means for receiving a data string 
including the code of waveform data and the 
execution frequency of each decoding unit 
grouped in accordance with the significance for 
maintaining the quality of the waveform data 
decoded from said code and outputting said 
code and said execution frequency; 

(b) estimating means for estimating the execu- 
tion time of each group in accordance with the 
processing time until obtaining a waveform 
after decoding said code and each of said exe- 
cution frequencies obtained from said data 
string; and 

(c) frequency reducing means for estimating 
the processing time required to decode a wave- 
form by using said execution frequency and 
said execution time, calculating the reduced 
number of execution frequencies of the groups 
in which said processing time does not exceed 
the time required to receive said code or the 
time from start of receiving said code up to 
start of receiving the next code (this is referred 
to as designated time) in accordance with each 
execution time output by said receiving means 
and each execution time output by said esti- 
mating means; wherein the time required for 
decoding is estimated and the execution fre- 
quency of each group is reduced so as to com- 
plete decoding within said designated time. 

71. A waveform data receiving method comprising the 
steps of: 

(a) receiving a data string including the code of 
waveform data and the execution frequencies 
of indispensable and dispensable processings 
for decoding and outputting said code and said 
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execution frequencies; 

(b) estimating the execution frequencies of said 
indispensable and dispensable processings in 
accordance with the processing time until 
obtaining a waveform after decoding said code 5 
and each of said execution frequencies 
obtained from said data string; and 

(c) estimating the processing time required to 
decode a waveform by using said execution 
frequency and said execution time, calculating w 
the reduced number of execution frequencies 

of said dispensable processing in which said 
processing time does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 15 
ing the next code (this is referred to as desig- 
nated time) in accordance with each execution 
frequency output by said receiving means and 
each execution time output by said estimating 
means, estimating the time required for decod- 20 
ing in accordance with each estimated execu- 
tion time, and reducing the execution frequency 
of said dispensable processing so as to com- 
plete decoding within said designated time. 

25 

72. A waveform data receiving apparatus comprising: 

(a) receiving means for receiving a data string 
including the code of waveform data and the 
execution frequencies of indispensable and 30 
dispensable processings for decoding and out- 
putting said code and said execution frequen- 
cies; 

(b) estimating means for estimating the execu- 
tion frequencies of said indispensable and dis- 3s 
pensable processings in accordance with the 
processing time until obtaining a waveform 
after decoding said code and each of said exe- 
cution frequencies obtained from said data 
string; and . 40 

(c) frequency reducing means for estimating 
the processing time required to decode a wave- 
form by using said execution frequency and 
said execution time and calculating the 
reduced number of execution frequencies of as 
said dispensable processing in which said 
processing time does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing the next code (this is referred to as desig- so 
nated time) in accordance with each execution 
frequency output by said receiving means and 
each execution time output by said estimating 
means; wherein 

the time required for decoding is estimated in 55 
accordance with each estimated execution 
time and the execution frequency of said dis- 
pensable processing is reduced so as to com- 
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plete decoding within said designated time. 

73. The video waveform data receiving method accord- 
ing to claim 69, wherein a video is output as said 
waveform data. 

74. The video waveform data receiving apparatus 
according to claim 70, wherein a video is output as 
said waveform data. 

75. The video waveform data receiving method accord- 
ing to claim 69, wherein (d) the execution time of 
each group obtained through estimation is output. 

76. The video waveform data receiving apparatus 
according to claim 70, wherein (d) the execution 
time of each group obtained by estimating means is 
output. 

77. The waveform data transmitting method according 
to claim 63, wherein 

(d) a data string including the execution time of 
each group is input, and 

(e) the execution frequency of each group is 
calculated in accordance with each execution 
time of said receiving means in order to com- 
plete decoding within the time required to 
transmit a code length decided by the designa- 
tion by a rate controller or the like. 

78. The waveform data transmitting apparatus compris- 
ing: 

(d) receiving means for inputting a data string 
constituted with the execution time of each 
group; and 

(e) deciding means for calculating the execu- 
tion frequency of each group in accordance 
with each execution time of said receiving 
means in order to complete decoding within the 
time required to transmit a code decided by the 
designation by a rate controller or the like. 

79. The video waveform data transmitting method 
according to claim 67, wherein 

(d) the execution time of each group is esti- 
mated in accordance with the processing time 
required to encode a video and said each exe- 
cution frequency; and 

(e) the processing time required to encode a 
video is estimated by using said execution time 
and the execution frequency of each group is 
calculated in which said processing time does 
not exceed the time usable to process a sheet 
of video determined in accordance with a 
frame rate given as the designation by a user. 
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80. A video waveform data transmitting apparatus 
according to claim 68, wherein 

(d) estimating means for estimating the execu- 
tion time of each group in accordance with the 5 
processing time required to encode a video 
and each execution time output by counting 
means; and 

(e) deciding means for estimating the process- 
ing time required to encode a video by using 10 
said execution time and calculating the execu- 
tion frequency of each group in which said 
processing time does not exceed the time usa- 
ble to process a sheet of video determined in 
accordance with a frame rate given as the des- is 
ignation by a user. 

81. The video waveform data transmitting method 
according to claim 63, wherein said counting result 
and the length of a code corresponding to wave- 20 
form data for a predetermined period are received 
when generation of said code is completed to trans- 
form the result and length into a data string. 
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time obtained from the length and transmission rate 
of said code. 

85. A waveform data receiving method for receiving the 
code of waveform data and decoding and output- 
ting the waveform, comprising the steps of: 

(a) constituting a data string including the des- 
ignation for selecting a processing unit having 
an execution time shorter than that of the 
encoding unit included in said code every 
encoding unit corresponding to a processing 
unit constituting the decoding process so that 
the processing time required to decode a wave- 
form does not exceed the time required to 
receive said code or the time from start of 
receiving said code up to start of receiving the 
next code (this is referred to as designated 
time); and 

(b) transmitting said data string to communi- 
cate to the transmitting side that a code for 
completing decoding within said designated 
time is transmitted. 



10 



15 



82. The video waveform data transmitting apparatus 
according to claim 63, wherein transforming means 
is included which receives said counting result of 
said counting means and the length of a code cor- 
responding to waveform data for a predetermined 
period when generation of said code is completed 
to transform the result and length into a data string. 

83. The waveform data receiving method according to 
claim 69, wherein a data string including a code 
corresponding to the waveform data for a predeter- 
mined period, the execution frequency of each 
decoding unit grouped in accordance with the sig- 
nificance for maintaining the quality of the wave- 
form data decoded from said code, and the length 
of said code is received and said code, execution 
frequency, and code length are output to reduce the 
execution frequency of dispensable processing so 
that the time required for decoding does not exceed 
a code transmission time obtained from the length 
and transmission rate of said code. 

84. The waveform data receiving apparatus according 
to claim 70, wherein receiving means is included 
which receives a data string including a code corre- 
sponding to the waveform data for a predetermined so 
period, the execution frequency of each decoding 
unit grouped in accordance with the significance for 
maintaining the quality of the waveform data 
decoded from said code, and the length of said 
code and outputs said code, execution frequency, 55 
and code length to reduce the execution frequency 

of dispensable processing so that the time required 
for decoding does not exceed a code transmission 



86. A waveform data receiving apparatus for receiving 
the code of waveform data and decoding and out- 
putting said waveform, comprising: 

(a) designated data constituting means for con- 
stituting a data string including the designation 
for selecting a processing unit having an execu- 
tion time shorter than that of the encoding unit 
included in said code every encoding unit cor- 
responding to a processing unit constituting the 
decoding process so that the processing time 
required to decode a waveform does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
referred to as designated time); and 

(b) transmitting means for transmitting said 
data string; wherein 

it is communicated to the transmitting side that 
a code for completing decoding within said 
designated time is transmitted. 

87. A waveform data transmitting method for encoding 
a waveform and outputting said code, comprising 
the steps of: 

(a) receiving a data string including the desig- 
nation for a processing unit to be selected for 
each processing unit constituting the encoding 
process; and 

(b) extracting said designation from said data 
string, encoding a waveform by using the 
processing unit specified in accordance with 
said designation, and outputting a coda 
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88. A waveform data transmitting apparatus for encod- 
ing a waveform and outputting said code, compris- 
ing: 

(a) receiving means for receiving a data string s 
including the designation for a processing unit 

to be selected for each processing unit consti- 
tuting the encoding process; and 

(b) extracting means for extracting said desig- 
nation from said data string; wherein 10 
a waveform is encoded by using the processing 
unit specified in accordance with said designa- 
tion to output a code. 

89. A waveform data receiving method for receiving the 75 
code of waveform data and decoding and output- 
ting a waveform, comprising the steps of: 

(a) counting the execution frequency of each 
processing unit constituting the waveform 20 
decoding process; 

(b) estimating the execution time for each 
processing unit in accordance with said execu- 
tion frequency and processing time required to 
decode a waveform; 25 

(c) constituting a data string including the des- 
ignation for selecting a processing unit having 
an execution time shorter than that of the 
encoding unit included in said code every 
encoding unit corresponding to the processing 30 

• unit constituting the decoding process so that 
the processing time required to decode a wave- 
form does not exceed the time required to 
receive said code or the time from start of 
receiving said code up to start of receiving the 35 
next code (this is referred to as designated 
time); and 

(d) transmitting said data string; wherein 

it is communicated to the transmitting method 
that a code for completing decoding within said 40 
designated time is transmitted. 

90. A waveform data receiving apparatus for receiving 
the code of waveform data and decoding and out- 
putting a waveform, comprising: 45 

(a) counting means, for counting the execution 
frequency of each processing unit constituting 
the waveform decoding process; 

(b) estimating means for estimating the execu- so 
tion time for each processing unit in accord- 
ance with said execution frequency and 
processing time required to decode a wave- 
form; 

(c) designated-data constituting means for con- 55 
strtuting a data string including the designation 

for selecting a processing unit having an execu- 
tion time shorter than that of the encoding unit 



included in said code every encoding unit cor- 
responding to the processing unit constituting 
the decoding process so that the processing 
time required to decode a waveform does not 
exceed the time required to receive said code 
or the time from start of receiving said code up 
to start of receiving the next code (this is 
referred to as designated time); and 
(d) transmitting means for transmitting said 
data string; wherein 

it is communicated to the transmitting side that 
a code for completing decoding within said 
designated time is transmitted. 

91. A video waveform data receiving method for receiv- 
ing the code of a video and decoding and outputting 
said video, comprising the steps of: 

(a) constituting a data string including the des- 
ignation for replacing the movement compen- 
sating method used to encode a video with the 
movement compensation processing having an 
execution time shorter than that of the move- 
ment compensation processing included in 
said code so that the processing time required 
to decode a video does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing the next code (this is referred to as desig- 
nated time); and 

(b) transmitting said data string; wherein 

it is communicated to the transmitting side that 
a code for completing encoding within said 
designated time is transmitted. 

92. A video receiving apparatus for receiving the code 
of a video and decoding and outputting said video, 
comprising: 

(a) designated-data constituting means for 
constituting a data string including the designa- 
tion for replacing the movement compensating 
method used to encode a video with the move- 
ment compensation processing having an exe- 
cution time shorter than that of the movement 
compensation processing included in said 
code so that the processing time required to 
decode a video does not exceed the time 
required to receive said code or the time from 
start of receiving said code up to start of receiv- 
ing the next code (this is referred to as desig- 
nated time); and 

(b) transmitting means for transmitting said 
data string; wherein 

it is communicated to the transmitting side that 
a code for completing encoding within said 
designated time is transmitted. 
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93. A video transmitting method for encoding a video 
and outputting said code, comprising the steps of: 

(a) receiving a data string including the desig- 
nation for the processing to be selected by 5 
using the movement compensating processing 
constituting the decoding process; and 

(b) extracting said designation from said data 
string; wherein 

encoding of a video is executed by using the 10 
movement compensating processing specified 
in accordance with said designation to output a 
code. 

94. A video transmitting apparatus for encoding a video is 
and outputting said code, comprising the steps of: 

(a) receiving means for receiving a data string 
including the designation for the processing to 

be selected by using the movement compen- 20 
sating processing constituting the decoding 
process; and 

(b) extracting means for extracting said desig- 
nation from said data string; wherein 

encoding of a video is executed by using the 25 
movement compensating processing specified 
in accordance with said designation to output a 
code. 
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o Information showing start position capable of 
processing data or not 

* Flag for random access (Random access flag), 
e.g. Intra- frame (I-picture) in the case of 
picture 

♦ Flag showing access unit (Access flag), 
e.g. Frame in the case of picture , GOB unit 



AL : Adaptation layer 
ES : Elementary stream 
PTS : Presentation • time • stamp 



Header 
information 
of data 



Data (Picture or sound for each frame) 




Information showing start position capable of 
processing data or not 

Information showing data reproducing time (PTS) 
Information showing data processing priority 
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F* X & . <4r- 

oTStTransport stream(Transmission packet) 



TS 




AL 



ES 



Communication 
header 



AL 



ES 



Communication 
header 

1— 

• Information showing start position capable of processing 
pieces of data or not 

• Identification number for showing data sequence(Sequence 
number) 

• Time concerned with transmission of pieces of data 



o Handling time stamp and marker bit 
(a) 



AL 



Communication 
header 

— i f 

Time stamp I 
PTS or not (Additional) 



ES 



(b) 



Communication 
header 

? 



AL 



T 



ES 




Time stamp for 

communication 

header 



Time stamp PTS or not (Additional) 



(c) 



Communication 
header 

? 

MarkerBit 



AL 



ES 



Substituted by 
AL flag (Additional) 



(d) 



AL 



ES 




MarkerBit of 

communication 

header 



Communication 
header 

t 

MarkerBit 

(It is interpreted that random access flag 
and access flag are present in AL.) 
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Data and 

control information 



H.223 or 
the like 



Group 
MUX 



UDP/TCP/RTP 



Intra-net and inter-net 

or the like 
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Fig. S 

Broadcast program transmitting procedure 
(Broadcast type and communication type including return channel) 
Transmitting side Receiving side 



ACK/Reject 



Transfer of data structure 

(LCN 0) : (*1) 



Transfer of corresponding data 
(From each port) : (*2) 



Are processing and 
reception possible? 
.Start decoding of 
data which can be 
decoded and display 
it. 



(Broadcast type (with no return channel)) 

Transmitting side Receiving side 

Transfer of program 
information and data structure 
(LCN O) : UDP(*3) 



Transfer of corresponding data 
(From each port) : UDP 



( * 1 ) Must be a system for detecting and retransmitting a packet 

loss like TCP. 
(*2) RTP/RTCP or TCP/IP 

(*3) Same data (picture or sound) or control information (broadcast 
program or data structure) is continuously repeatedly 
transmitted. A packet is detected and sequence is kept at a 
receiving terminal in accordance witha sequence number. (To be 
used in a local closed region. Traffic becomes too large.) 
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i & . 1 O ( a ) 

Receiving terminal 



Control information 
or data 



Program or data 
to be required 



Flag, counter, 
or timer showing 
point of time to be 
required 



Main . 
looking-listening - 
section 






Auxi 
looking-i 
sec 


liary 

listening 

tion 



Storing 
section 



Output 
section 



f i « . X O ( b ) 

Receiving terminal 



Control information 
or data 

»> 



Main 

looking-listening 
section 



Caption 
broadcast-program 
receiving section 



Output 
section 



Storing 
section 
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F i « . 11(a) 
<Hierarchical image of object> 




transmission image of object> 
<1. Broadcast type> 




<2. Communication type> 

RTP/RTCP (Program ID of each logical 
channel is fixed. ) 





< — — . _ _ — . — 




Terminal 




Terminal 






A 




B 




< - > 





LCNO (control) 
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Fig. 11(b) 



-Capability exchange definitions(original from H.245) 



Terminal CapabilitySet 
{ 

sequenceNumber 



::=SEQUENCE 
SequenceNumber, 



multiplexCapability 
capabilityTable 

capabilityDescriptors 

mpeg4Capability 



MultiplexCapability OPTIONAL 
SET SIZE0..256) OF Capability 
TableEntryOPTIONAL, 
SET SIZE0..256) OF Capability 
DescriptorOPTIONAL, 
MPEG4Capability OPTIONAL. 
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-MPEG4 Capability definitions 



MPEG4Capability 



{ 



sequenceNumber 

NumberOfProcessObject 

{ 

MaxNumberOfVideo 
MaxNumberOf Sounds 
MaxNumberOfMux 

} 

reconfigurationALCapability 



} 



MPEG4CapabilityAck 



{ 



sequenceNumber 



} 

MPEG4CapabilityReject 
{ 

sequenceNumber 
NumberOfProcessObject 

maxNumberOfVideo 

maxNumberOf Sounds 

MaxNumberOfMux 

} 

reconf i gurationALCapabitity 



::=SEQUENCE 

SequenceNumber, 
SEQUENCE 

INTEGER(0..1023), 

INTEGER(0..1023). 
INTEGER(0..1023), 
BOOLEAN, 

::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
SEQUENCE 

MaxNumberOfVideo, 

MaxNumberOf Sounds 

nnaxNumberOfMux, 

BOOLEAN, 
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i & . 13(a) 



—Group MUX definitions 



CreateGroupMux 
{ 

sequenceNumber 
Group MuxID 
lanportNumber 

} 

CreateGroupMuxAck 



} 



sequenceNumber 



Cr eateGroup MuxRe ject 

sequenceNumber 

cause 

I 



::=SEQUENCE 

SequenceNumber, 

INTEGER0D..1O23), 
LANPortNumber, 



::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 
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Fig 



13(b) 



DestoryGroupMux 



{ 



sequenceNumber 
GroupMuxID 



Destory Group Mux Ack 



I 



sequenceNumber 



Destory Group MuxReject 
{ 

sequenceNumber 

cause 

{ 



} 



::=SEQUENCE 

SequenceNumber, 
INTEGER(0..1023). 



::=SEQUENCE 

SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 
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13(c) 



PortNumberStructure 
{ 



::=SEQUENCE 



sequenceNumber SequenceNumber, 
lanPortNumber LANPortNumber, 
numberOfLogicalNumber INTEGER0..15), 
SEQUENCE SIZE0..15) OF PortStructureElement, 



PortStructureElement 



{ 



logicalPortNumber 



} 

PortNumberStructureAck 



sequenceNumber 



} 

PortNumberStructureReject 

sequenceNumber 
cause 

{ 



::=SEQUENCE 
LogicalPortNumber, 

::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 
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Fig. 1^ 



-Logical channel signalling definitions(original from H.245) 
-MPEG4 Object Create Operation(for LANPortNumber) 



fowardLogicalChannelNumber LogicalChannelNumber, 

fowardLogicalChannelParameters SEQUENCE 
{ 



I 

h222LogicaiChamelParafneters H222LogicalChannelParameters, 

h223LogicalChamelParameters H223LogicalChannelParameters f 

v76LogicalChannelParameters v76LogicalChannelParameters, 

• * • 9 

h2250LogicalChannelParameters H2250LogicalChamelParameter^ 
h223AnnexALogicalChannelParameters 
H223AnnexALogicalChannelParameters 
MPEG4LogtalChamelParameters MPEG4LogicalChanelParameters, 



OpenLogicalChannel 



::=SEQUENCE 



portNumber 
dataType 

multiplexParameters 



INTEGER(CL65535)0PTI0NAL 
DataType. 
CHOICE 



— r 
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MPEG4LogicalChannelFarameters ::=SEQUENCE 



-H.225BASE 
LANportNumber 
ProgramID 
ProgramName 



BroadcastChannelProgram 



INTEGER(0.. 65535), 
INTEGER(0..255), 

0CTETSTRiNG(SIZE(128)), 



::=SEQUENCE 



sequenceNumber SequenceNumber, 
numberOf ChannelNumber INTEGER(0..1023), 
SEQUENCE SIZE(1..1023) OF MPEG4LogicalChannelParameters 



ChangeLogicalChannelAttribute 

sequenceNumber 
lanportNumber 
ProgramID 



ChangeLogicalChannelAttributeAck 
sequenceNumber 

^ngeLc^icaLChanneLAttributeReject 

sequenceNumber 
cause 
{ 



::=SEQUENCE 

SequenceNumber 
LANPortNumber, 
INTEGER(0..255), 



::=SEQUENCE 
SequenceNumber, 

::=SEQUENCE 

SequenceNumber, 
CHOICE 
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-MPEG4 Object Class definition 



MPEG4 Object Class definition 



:;=SEQUENCE 



sequenceNumber SequenceNumber, 
-ProgramID INTEGER(0..255), 
NumberOfObjectsList INTEGER(0..1023), 
SEQUENCE SIZE(1..1023) OF ObjectStructureElement 



Ob jectStructureElement 



0 



SSRC 

LANPortNumber 



ScrambleFlag 
CGDOffset 
MediaType 



::=SEQUENCE 

INTEGER(0..16777215), 
INTEGER0O24.5OOO), 

— forRPT(Video&Sound) 
BOOLEAN, 
INTEGERC0..255), 
INTEGERC0..255), 



) 



MPEG4 Object Class defiiitionAck 



{ 



sequenceNumber 



} 



{ 



sequenceNumber 

cause 

{ 



::=SEQUENCE 
SequenceNumber, 



MPEG4 Object Class defirttionReject ::=SEQUENCE 



SequenceNumber, 
CHOICE 
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P i g.' 16(b) 



-Adaptation Layer Reconfiguration Request definitions 



ALReconfiguration 
{ 

sequenceNumber 
RandomAccessFlagMaxBit 
PresentationTimeStampsMaxBit 
CGDPriorityMaxBit 



::=CHOICE 

SequenceNumber, 

INTEGER(0...2). . 
INTEGER(0...32), 

INTEGER(0...8), 

~f orVideo and Sound 



-Adaptation Layer Reconfiguration Response definitions 



ALReconfigurationAck 
sequenceNumber 

) 

ALReconfigurationReject 



sequenceNumber 

cause 

( 



::=SEQUENCE 
SequenceNumber, 

—SEQUENCE 

SequenceNumber, 
CHOICE 



<Relatlon between AL, ES, and RTP> 
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-Setup Program and Data Request definitions 



Setup Request 



{ 



sequenceNumber 
SSRC IMEGER(0. .16777215)2*32, 



::=CH0ICE 
SequenceNumber, 



Logical Channel Number, 
setupitem 

{ 

executeProgramNumber 
data Number 

exeeuteCommandNumber 

U 

nofitycounter 
{ 



flag 

counter 
timer 



INTEGER(1024...5000), 
CHOICE 

INTEGER(0...255), 
INTEGER(0...255), 
INTEGER(0...255), 

CHOICE 

BOOLEAN 

INTEGER(0...255), 
INTEGERC0...255), 
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-control and AL attribute definitions 



ControlALdefinition 
{ 

sequenceNumber 
AL 



{ 



RandomAccessFlagUse 

PresentationTimeStampUse 

CGDPriorityUse 



::=CH0ICE 



SequenceNumber, 
CHOICE 

BOOLEAN, 
BOOLEAN, 
BOOLEAN,- 
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classES_header{ . 

uint(4) headerlD; 
uint(24) bufferSizeES; 
uint(1) useTimeStamps; 



} 



u i n t ( 1 6 ) sequenceNumberMaxBit; 
u i nt ( 1 ) useHeaderExtension; 
if (useHeaderExtensionM 



} 

uint(3) reserved: 



uint(l) accessUintStartFlag; 

uint(1) randomAccessPointFlag; 

uintd) OCRsetFlag; 

u i nt (4) degradationPriorityMaxBit; 
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Fig 



19(b) 



-Adaptation Layer PDU header configuration Request and Command definition 



AL configuration 



{ 



sequenceNumber 

defaultHeaderConfiguration 

headerlD 

MPEG4ALPDUHeaderConfig 



{ 



accessllintStartFlag 

randomAccessPointFlag 

OCRsetFlag 

degradationPriorityMaxBit 



::=SEQUENCE 

SequenceNumber, 
BOOLEAN, 
INTEGERC0..4). 
SEQUENCE 

BOOLEAN, 
BOOLEAN, 
BOOLEAN, 
INTEGER(0..4), 
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Processing at receiving terminal under overload(Common to dynamic 
picture and sound) 

Thread for processing sound at system level is previously set it's 
processing priority to a value higher than that of thread for 
processing picture. 

Step 101 
Value of resolution of 
priority to be added to 
» frame, CGDoff set(Can be 
determined in accordance 
with receiving-terminal 
performance) is transmitted 
as control information. 



Frame_skipped 
. =FLASE 



Set priority of 

frame to be 
disused to value 

larger than 
(CutOffPriority) 
"maxPriority". 



Step 103 



Step 102 



CutOffPriority++ 




CutOffPriority=0 



CutOffPriority 
=maxPriority 




YES 



Step 104 



Frame_skipped=FLASE 



Z 



Frame_skipped 
=TRUE 



Deliver data to decoder. 
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Fig. 2 7(a) 

4031 



4032 



/ 




/ 




Priority adding 








Priority deciding 






section 








section 











4101.4201 Picture-sound encoder 

4102 . 4202 Picture-sound decoder 



F* i « . 2 T{ b ) 



Information 
for priority 






j 












i 





Picture frame 
or sound frame 



Dividing 




Communication header 



Pay load 
(Divided data) 
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RTP header 



H.263 payload header 



H.263 bitstream 



oMode A: GOB. picture boundary 

Presence or absence of mode or PB, start 

and end positions of bit stream, and Core 

execution timing states of options of — *-infor- 

resolution, frame type, and H.263 mation 

DBQUANT,TR(f or B frame), _^ To be set when 
TR(f or P frame) PB frame is present 

oMode B: MB boundary without PB 

Core information for Mode A 

Information for quantization value (GQUANT) , GOB number, 
absolute address of first MB in GOB, and movement vector 
(Horizontal and vertical directions) 

oMode C: MB boundary with PB 
Information for Mode B 

DBQUANT,TR(f or B f rame),TR(f or P frame) 



Relating of communication payload 



Transmitting- 
side terminal 



Mode A + Layer No- +J|JJ*gL 

tion 

> 

Configuration information 



Receiving - 
side 

terminal 
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Data 



Priority at application level 



Dividing 
of data 



_. \ 
Da 


f 

ta 




Da 


r 

ta 



Priority in IP level 



StreamPriority 
FramePriority 



Relating 



Priority of IP 



Priority in data 



Available range 



StreamPriority 


0 


~ 3 


FramePriority 


0 


~ 5 


IPV6 


8 


~ 15 



[0- - -15] 

j^^x^ Mapping of part 



15 - [8- • -15] 



( Lowes t ) ( Maximum 
priority) 
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4 o 



Start 



Input picture. 



V 



801 




Intra 




Movement 
compensation 




i 


i 


i/ 805 


DCT 




t 


/806 


VLC 









807 



Processing 
tTf every macrobloc> 
NO completed? 



80 A 



YES 



Generate and multiplex 
data string. 

I 



808 
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F i 



4- 1 



Start 

9 



Initialize execution 
time (a_i) of each 
element. 



901 



• Input multiplexed 
data and measure time 
required to input 
multiplexed data. 

I ~ 



902 



Start of decoding routine 



Output variable-length 
code and data string. 



Fetch execution 
frequency from data 

string and set 
execution frequency 
to x_i. 



•903 



.904 



Calculate execution 

frequency in 
accordance with x_J 
and a i. 



I 



.905 



Start time 
measurement. 



906 



To decoding 
routine 

i 



VLD 

T~ 



IDCT 



910 
911 




Movement 
compensation 



Complete time 
measurement. 

i 



Estimate execution 
time of each element 
and update a_i. 

I 




914 



End of decoding routine 
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1004 



Receive data string 
and fetch and set 
execution time of 
each processing. 



806 



Generate and 
multiplex data 
string. 
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NO 



Start 

9 



Initialize each 
execution time. 



Input picture. 



Divide macroblock. 



± 



/1001 

801 
/802 



Decide intra/inter on 
every macroblock. 

I 



Decide execution 
frequency. 



V 



1002 



1003 




Intra 



804 



Movement 
compensation 



DCT 



/ 



VLC 



/ 



805 



806 



807 



Processing 
"of every macroblock" 
completed? 



YES 



93 



EP 0 905 976 A1 



4 3 



-1101 



Start 

9 



Intialize execution 
time a_i of each 
proce ssing. 

1 



Input frame rate. 



1102 



T 



1104 



Estimate 
execution time of 
each processing 
and -update a_i. 



808 



Generate and 
multiplex data 
string. 



Input picture. 



801 



Divide macroblock. 



V 



i 



802 



Decide intra/inter on 
every macroblock. 



1 



Decide execution 
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